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WE CLAIM : 

1 1 . A method for obtaining a relatively high dynamic range image of a scene using a 

2 relatively low dynamic range image sensor exposed to incident light from the scene for capturing 

3 an image thereof, the image sensor having a multiplicity of light-sensing elements in an array, 

4 each light-sensing element having a particular one of a plurality of sensitivity levels to incident 

5 light in accordance with a predetermined sensitivity pattern for the array of light-sensing 

6 elements and having a response function, each light-sensing element being responsive to incident 
=,□7 light from the scene for producing a captured image brightness value at a corresponding one of a 

multiplicity of pixel positions of a pixel position array, whereby each one of the multiplicity of 

;:£9 pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light- 

^TO sensing elements, the method comprising the steps of: 

r§l (a) estimating respective off-grid brightness values at a multiplicity of off-grid positions 

' ] S2 from respective captured image brightness values at pixel positions of the pixel position array, 

-f : 3 the off-grid positions being located at respective interstices of the pixel position array and 

14 forming a regular off-grid position array; and 

15 (b) deriving pixel brightness values of an output image from the estimated off-grid 

16 brightness values. 

1 2. The method of claim 1, wherein the array of light-sensing elements of the image sensor 

2 is a linear array for capturing a line image, and the pixel position array is a linear array having 
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3 the multiplicity of pixel positions at respective regularly spaced pixel positions in a pixel row so 

4 as to define a linear captured image brightness value array. 

1 3 . The method of claim 1 , wherein the array of light-sensing elements of the image 

2 sensor is a linear array and the pixel position array is a two-dimensional array haivng pixel 

3 positions at respective intersections of a plurality of regularly spaced pixel rows and a plurality 

4 of regularly spaced pixel columns , the pixel rows being orthogonal to the pixel columns, and 
;i5 wherein the linear array of light-sensing elements of the image sensor is sequentially exposed to 
= n6 successive regularly spaced linear regions of the scene in multiple exposures, each one of the 
= I7 exposures producing a respective linear captured image brightness value array, the respective 

linear captured image brightness arrays produced by the multiple exposures being concatenated 

;39 to form a two-dimensional captured image brightness value array having respective captured 

W image brightness values at the pixel positions of the pixel position array. 

1 4. The method of claim 1, wherein the image sensor has a two-dimensional array of 

2 light-sensing elements and the pixel position array is a two-dimensional array having the 

3 multiplicity of pixel positions at respective intersections of a plurality of regularly spaced pixel 

4 rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the 

5 pixel columns. 
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1 5. The method of claim 4, wherein the off-grid positions of the off-grid position array are 

2 located at respective intersections of a plurality of regularly spaced intermediate rows and a 

3 plurality of regularly spaced intermediate columns, each one of the intermediate rows being 

4 parallel to the pixel rows and extending between a respective adjacent pair of the pixel rows, 

5 each one of the intermediate columns being parallel to the pixel columns and extending between 

6 a respective adjacent pair of the pixel columns. 

ol 6. The method of claim 4, wherein the off-grid positions of the off-grid position array are 

UT2 located at respective intersections of a plurality of regularly spaced intermediate rows and the 

j: ~3 pixel columns, and at respective intersections of a plurality of regularly spaced intermediate 

"^4 columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows 

q5 and extending between a respective adjacent pair of the pixel rows, each one of the intermediate 

TU6 columns being parallel to the pixel columns and extending between a respective adjacent pair of 

;:;?7 the pixel columns. 

1 7. The method of claim 3, wherein the off-grid positions of the off-grid position array are 

2 located at respective intersections of a plurality of regularly spaced intermediate columns and the 

3 pixel rows, each one of the intermediate columns being parallel to the pixel columns and 

4 extending between a respective adjacent pair of the pixel columns. 
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1 8. The method of claim 2, wherein the off-grid positions of the off-grid position array are 

2 located between respective adjacent pairs of pixel positions of the linear pixel position array. 

1 9. The method of claim 1 , wherein the step of deriving the output image from the 

2 estimated off-grid brightness values comprises using the estimated off-grid brightness values 

3 directly as the pixel brightness values of the output image. 

Ql 10. The method of claim 1 , wherein the step of deriving pixel brightness values of the 

lR2 output image from the estimated off-grid brightness values comprises resampling the estimated 

r Z3 off-grid brightness values to derive respective resampled on-grid brightness values at the pixel 

JM- positions of the pixel position array, the respective resampled on-grid brightness values at the 

□5 pixel positions of the pixel position array being the pixel brightness values of the output image. 

Izl 1 1 . The method of claim 5 5 wherein the predetermined sensitivity pattern of the array of 

2 light-sensing elements has four different predetermined sensitivity levels and corresponds to a 

3 pixel position array having repetitive groups of four nearest neighbor pixel positions, each one of 

4 the four pixel positions in each group corresponding to a different one of the four predetermined 

5 sensitivity levels in a common predetermining positional order. 
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1 12. The method of claim 6, wherein the predetermined sensitivity pattern of the array of 

2 light-sensing elements has first and second predetermined sensitivity levels, and wherein 

3 adjacent pixel positions in each one of the pixel rows and adjacent pixel positions in each one of 

4 the pixel columns correspond to different ones of the first and second predetermined sensitivity 

5 levels. 

1 13. The method of claim 7, wherein the predetermined sensitivity pattern of the array of 

;;j2 light-sensing elements have first and second predetermined sensitivity levels, and adjacent pixel 

1113 positions in each one of the pixel rows correspond to different ones of the predetermined first and 

^4 second sensitivity levels, and adjacent pixel positions in each one of the pixel columns 

m 

^5 correspond to the same one of the predetermined first and second sensitivity levels. 

TUl 14. The method of claim 1, wherein the step of estimating respective off-grid brightness 

;;^2 values at the off-grid positions of the off-grid position array from respective captured image 

3 brightness values at the pixel positions of the pixel position array comprises the steps of: (1) 

4 estimating a sensitivity level off-grid brightness value for each different one of the sensitivity 

5 levels at each one of the off-grid positions using only captured image brightness values at pixel 

6 positions corresponding to the different one of the sensitivity levels; and (2) combining the 

7 sensitivity level off-grid brightness value estimated for each different one of the sensitivity levels 
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at each one of the off-grid positions to derive a respective estimated off-grid brightness values at 
each one of the off-grid positions. 

15. The method of claim 14, wherein the step of combining the sensitivity level of 
brightness value for each different one of the sensitivity levels at each one of the off-grid 
positions comprises adding the sensitivity level off-grid brightness values estimated for each 
different one of the sensitivity levels at each one of the off-grid positions to derive the respective 
estimated off-grid brightness value at each one of the off-grid positions. 

16. The method of claim 15, wherein the step of deriving pixel brightness values of the 
output image from the estimated off-grid brightness values comprises interpolating the estimated 
off-grid brightness values to derive respective interpolated on-grid brightness values at the pixel 
positions of the pixel position array, and compensating each one of the interpolated on-grid 
brightness values by the inverse of a combined response function of the light-sensing elements to 
obtain the pixel brightness values of the output image, the combined response function being the 
sum of weighted response functions of light-sensing elements having different ones of the 
sensitivity levels, each one of the weighted response functions being a response function of a 
light-sensing element having a respective one of the sensitivity levels multiplied by a 
predetermined weighting factor for the respective one of the sensitivity levels. 
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1 17. The method of claim 16, wherein the step of compensating each one of the 

2 interpolated on-grid brightness values by the inverse of the combined response function of the 

3 light-sensing elements comprises deriving from each one of the interpolated on-grid brightness 

4 values a respective index for a lookup table memory storing lookup table data representing the 

5 inverse of the combined response function of the light-sensing elements, retrieving from the 

6 lookup table memory a compensated interpolated on-grid brightness value corresponding to each 

7 index, and providing the compensated interpolated on-grid brightness values as the pixel 
1:38 brightness values of the output image. 

r ~l 18. The method of claim 14, wherein the step of combining the sensitivity level off-grid 

; g2 brightness value for each different one of the sensitivity levels at each one of the off-grid 

C33 positions comprises the steps of: (1) multiplying the sensitivity level off-grid brightness value for 

i w4 each different one of the sensitivity levels at each one of the off-grid positions with a respective 

;S5 predetermined weighting factor corresponding to the different one of the sensitivity levels to 

6 derive a weighted sensitivity level off-grid brightness value for each different one of the 

7 sensitivity levels at each one of the off-grid positions; and (2) adding the weighted sensitivity 

8 off-grid level brightness value for each different one of the sensitivity levels at each one of the 

9 off-grid positions to derive the respective estimated off-grid brightness value at each one of the 
1 0 off-grid positions. 
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1 19. The method of claim 1 8, wherein the step of deriving the pixel brightness values of 

2 the output image from the estimated off-grid brightness values comprises interpolating the 

3 estimated off-grid brightness values to derive respective interpolated on-grid brightness values at 

4 the pixel positions of the pixel position array, and compensating each one of the interpolated on- 

5 grid brightness values by the inverse of a combined response function of the light-sensing 

6 elements to derive the pixel brightness values of the output image, the combined response 

7 function being the sum of weighted response functions of the light-sensing elements having 
q8 different ones of the sensitivity levels, each one of the weighted response functions being a 
j =R9 response function of a light-sensing element having a respective one of the sensitivity levels 
|S I0 multiplied by a predetermined weighting factor for the respective one of the sensitivity levels. 

□1 20. The method of claim 19 wherein the step of compensating each one of the 

I y 2 interpolated brightness values by the inverse of a combined response function of the light- 

23 sensing elements comprises using each one of the interpolated on-grid brightness values to derive 

4 a respective index for a lookup table memory storing data representing the inverse of the 

5 combined response function of the light-sensing elements, retrieving from the lookup table 

6 memory a respective compensated interpolated on-grid brightness value corresponding to each 

7 index, and providing the compensated interpolated on-grid brightness values as the pixel 

8 brightness values of the output image. 
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1 21 . The method of claim 1, wherein the step of estimating respective off-grid brightness 

2 values at the off-grid positions of the off-grid position array from respective captured image 

3 brightness values at the pixel positions of the pixel position array comprises the steps of: (1) 

4 estimating a sensitivity level off-grid brightness value for each different one of the sensitivity 

5 levels at each one of the off-grid positions using only the captured image brightness values at 

6 pixel positions corresponding to the different one of the sensitivity levels; (2) combining the 

7 sensitivity level off-grid brightness value estimated for each different one of the sensitivity levels 
□8 at each one of the off-grid positions to derive a combined sensitivity level off-grid brightness 
Lf§9 value at each one of the off-grid positions; and (3) compensating the combined sensitivity level 
|s f 0 off-grid brightness value at each one of the off-grid positions by the inverse of a combined 

" = 11 response function of the light-sensing elements to derive a respective estimated off- grid 

ii %2 brightness value at each one of the off-grid positions, the combined response function being the 

sum of weighted response functions of light-sensing elements having different ones of the 

|:|4 sensitivity levels, each one of the weighted response functions being a response function of a 

15 light-sensing element having a respective one of the sensitivity levels multiplied by a 

16 predetermined weighting factor for the respective one of the sensitivity levels. 

1 22. The method of claim 21, wherein the step of compensating the combined sensitivity 

2 level off-grid brightness value at each one of the off-grid positions by the inverse of the 

3 combined response function of the light-sensing elements comprises using the combined 
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4 sensitivity level off-grid brightness value at each one of the off-grid positions to derive a 

5 respective index for a lookup table memory storing data representing the inverse of the combined 

6 response function of the light-sensing elements, retrieving from the lookup table a compensated 

7 combined sensitivity level off-grid brightness value corresponding to each index, and providing 

8 the compensated combined sensitivity level off-grid brightness values retrieved from the lookup 

9 table as the respective estimated off-grid brightness values at the off-grid positions. 

gl 23. The method of claim 21 , wherein the step of combining the sensitivity level off-grid 

n 

fU brightness value estimated for each different one of the sensitivity levels at each one of the off- 

3 grid positions comprises adding the sensitivity level off-grid brightness value for each different 

^4 one of the sensitivity levels at each one of the off-grid positions to derive the respective 

g5 combined sensitivity level off-grid brightness values at the off-grid positions. 

if 1 24. The method of claim 21, wherein the step of combining the sensitivity level off-grid 

2 brightness value estimated for each different one of the sensitivity levels at each one of the off- 

3 grid positions comprises the steps of: (1) multiplying the sensitivity level off-grid brightness 

4 value estimated for each different one of the sensitivity levels at each one of the off-grid 

5 positions by a respective predetermined weighting factor corresponding to the different one of 

6 the sensitivity levels to derive a weighted sensitivity level off-grid brightness value for each 

7 different one of the sensitivity levels at each one of the off-grid positions; and (2) summing the 
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8 weighted sensitivity off-grid level brightness value for each different one of the sensitivity levels 

9 at each one of the off-grid positions to derive the combined sensitivity level off-grid brightness 
10 value at each one of the off-grid positions. 

1 25. The method of claim 14 or 21 , wherein the step of estimating the sensitivity level off- 

2 grid brightness value for each different one of the sensitivity levels at each one of the off-grid 

3 positions comprises the steps of: (1) masking the pixel positions but leaving unmasked pixel 
rA positions corresponding to the different one of the sensitivity levels; and (2) estimating the 

1115 sensitivity level off-grid brightness value for the different one of the sensitivity levels at the one 

of the off-grid positions to be equal to the captured image brightness value at the one of the 

^7 unmasked pixel positions nearest the one of the off-grid positions. 

rUl 26. The method of claim 24, wherein the array of light-sensing elements of the image 

;;f2 sensor is a linear array and the pixel position array is a linear array having pixel positions at 

3 respective regularly spaced pixel positions in a pixel row and the off-grid positions of the off- 

4 grid position array are each located between respective pairs of adjacent pixel positions of the 

5 linear pixel position array, and wherein the step of estimating a sensitivity level off-grid 

6 brightness value for each different one of the sensitivity levels at each one of the off-grid 

7 positions comprises providing as the sensitivity level off-grid brightness value for each different 

8 one of the sensitivity levels at each one of the off-grid positions the captured image brightness 
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9 values at pixel positions corresponding to the different one of the sensitivity levels that is the 

10 nearest neighbor to the one of the off-grid positions. 

1 27. The method of claim 26, wherein the predetermined sensitivity pattern of the light- 

2 sensing elements has first and second predetermined sensitivity levels, and adjacent pixel 

3 positions of the liner pixel position array correspond to different ones of the first and second 

4 predetermined sensitivity levels, and wherein the step of multiplying the sensitivity level off-grid 
p5 brightness value for each different one of the sensitivity levels at each one of the off-grid 

U r i6 positions by a respective predetermined weighting factor corresponding to the different one of 

i= l!7 the sensitivity levels to derive a weighted sensitivity level off-grid brightness value for each 

: "g8 different one of the sensitivity levels at each one of the off-grid positions, and the step of 

o9 summing the weighted sensitivity level off-grid brightness value for each different one of the 

| WO sensitivity levels at each one of the off-grid positions to derive the combined sensitivity level off- 

21 grid brightness value at each one of the off-grid positions is expressed as 

12 B(x') = W x I c {x' - 0.5) + W 2 I c (x ( + 0.5), 

13 where B(x') is the combined sensitivity level off-grid brightness value at off-grid position x', Wj 

14 is the predetermined weighting factor for the sensitivity level corresponding to pixel position 

15 (x' - 0.5), W 2 is the weighting factor for the sensitivity level corresponding to pixel position 

16 (x' + 0.5), I c (x' - 0.5) is the captured image brightness value at pixel position (x' - 0.5) and 



NY02;24 1764.1 



-94- 




FILE NO. A32708-070050.1125 

17 I c (x' + 0.5) is the captured image brightness value at pixel position (x' + 0.5), and wherein the 

1 8 combined response function of the light-sensing elements is expressed as 

19 S(E) = Wf x (E) + W 2 P 2 {E), 

20 where Pj(E) is the radiometric response function of a light- sensing element having the sensitivity 

21 level corresponding to pixel position (x' - 0.5) and P 2 (E) is the radiometric response function of 

22 a light-sensing element having the sensitivity level corresponding to pixel position (x' + 0.5). 

q1 28. The method of claim 1, wherein the step of estimating respective off-grid brightness 

UT2 values at the off-grid positions of the off-grid position array from respective captured image 

' r j3 brightness values at the pixel positions of the pixel position array comprises bi-linear sampling 

^4 with interpolation of the captured image brightness values at the pixel positions. 

FUl 29. The method of claim 1, wherein the step of estimating respective off-grid brightness 

;:^2 values at the off-grid positions of the off-grid position array from respective captured image 

3 brightness values at the pixel positions of the pixel position array comprises bi-cubic sampling 

4 with interpolation of the captured image brightness values at the pixel positions. 

1 30. The method of claim 1 , wherein the step of estimating respective off-grid brightness 

2 values at the off-grid positions of the off-grid position array from respective captured image 

3 brightness values at the pixel positions of the pixel position array comprises estimating a a 
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4 respective off-grid brightness value at each individual one of the off-grid positions by comparing 

5 a respective captured image brightness value at each one of a predetermined number of pixel 

6 positions that are nearest neighbors to the individual one of the off-grid positions with a 

7 predetermined low threshold value and a predetermined high threshold value, computing the sum 

8 of the nearest neighbor pixel position captured image brightness values that are greater than the 

9 predetermined low threshold value and less than the predetermined high threshold value, and 
10 dividing the sum by the number of nearest neighbor pixel position brightness values included in 

=:|1 the sum. 

! '~ 1 31. The method of claim 1 , wherein the step of estimating off-grid brightness values at 

~'n2 the off-grid positions of the off-grid position array from respective captured image brightness 

□3 values at the pixel positions of the pixel position array comprises estimating a respective off-grid 

U4 brightness value at each individual one of the off-grid positions by comparing a respective 

compensated captured image brightness value at each one of a predetermined number of pixel 

6 positions that are nearest neighbors to the individual one of the off-grid positions with a 

7 predetermined low threshold value a predetermined high threshold value, computing the sum of 

8 the compensated captured image brightness values at the nearest neighbor pixel positions that are 

9 greater than the predetermined low threshold value and less than the predetermined high 

10 threshold value, and deriving a respective estimated off-grid brightness value at the individual 

1 1 one of the off-grid positions by dividing the sum by the number of compensated captured image 
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12 brightness values at nearest neighbor pixel positions, the respective compensated captured image 

1 3 brightness value at each one of the nearest neighbor pixel positions being the captured image 

14 brightness value at the one of the nearest neighbor pixel positions compensated by the inverse of 

15 the response function of a light-sensing element having a sensitivity level corresponding to the 

16 one of the nearest neighbor pixel positions. 

1 32. The method of claim 31, wherein the respective compensated captured image 

□2 brightness value at each one of the nearest neighbor pixel positions is derived by using the 

y!3 captured image brightness value at the one of the nearest neighbor pixel positions to derive an 

? Z4 index for a lookup table memory storing data representing the inverse response functions of 

Jh5 light-sensing elements having different ones of the plurality of sensitivity levels, and retrieving 

□6 from the lookup table memory a compensated captured image brightness value corresponding to 

i"U7 the index from lookup table data representing the inverse of the response function of a light- 

!iw 8 sensing element having the sensitivity level of the one of the nearest neighbor pixel positions. 

1 33. The method of claim 5, wherein each one of the intermediate rows extends medially 

2 between a respective adjacent pair of the pixel rows and each one of the intermediate columns 

3 extends medially between a respective adjacent pair of the pixel columns, and wherein the step of 

4 estimating respective off-grid brightness values at the off-grid positions of the off-grid position 

5 array from respective captured image brightness values at the pixel positions of the pixel position 
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6 array comprises estimating the brightness value at each one of the off-grid positions (x',y'), 

7 where x' has values from 1 to xSize- 1, y' has values from 1 to ySize- 1, and xSize and ySize are 

8 the dimensions of the captured image in the direction of the pixel rows and the pixel columns, 

9 respectively, the step of estimating respective off-grid brightness values at each one of the off- 

10 grid positions (x',y') comprising: 

1 1 (i) comparing a respective brightness value I(u,v) at each one of sixteen pixel 

12 positions (u,v) that are nearest neighbors to the one of the off-grid positions (x',y') with a 

r|3 predetermined low threshold value and a predetermined high threshold value, where u has values 

==04 from x' - 5 to x' + 0.5 and v has values from y' - 0.5 to y' + 0.5; 

%5 (ii) for each one of the brightness values I(u,v) at the sixteen nearest neighbor 

. "^6 pixel positions (u,v) that is greater than the predetermined low threshold value and less than the 

!;i|7 predetermined high threshold value, computing indices i and j of a 4 x 4 cubic interpolation 

[ IJ8 kernel (f) using the relations i = x'-u- 1 .5 and j = y v- 1 .5, and computing the quantities 

4>, I(u>v) 



2 J_Ji_ 

iJ E cD; 

ab 



ab 



19 and (J>^., where (J>^ is the value of the 4 * 4 cubic interpolation kernel (|> at indices m and n; 

20 (iii) adding the quantities 



iJ E <t> 2 
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21 computed in step (ii) to derive a first sum s; 

22 (iv) adding the quantities <t>^. computed in step (ii) to derive a second sum w; and 

23 (v) computing the estimated brightness value at the one of the off-grid positions 

24 (x',y') by dividing the first sum s by the second sum w. 

1 34. The method of claim 33, wherein the brightness value I(u,v) at nearest neighbor pixel 

2 position (u 5 v) compared with the predetermined low threshold and the predetermined high 
pt3 threshold value in step (i) is the captured image brightness value at pixel position (u,v). 

H4 35. The method of claim 33, wherein the brightness value I(u,v) at nearest neighbor pixel 

position (u,v) compared with the predetermined low threshold value and the predetermined high 

i==|3 threshold value in step (i) is the captured image brightness value at pixel position (u,v) 

ry4 compensated by the inverse of a response function of a light-sensing element having the 

y 5 sensitivity level corresponding to pixel position (u 5 v). 

1 36. The method of claim 35, wherein the captured image brightness value at each one of 

2 the nearest neighbor pixel positions (u,v) is compensated by the inverse of the response function 

3 of a light-sensing element having the sensitivity level corresponding to one of the nearest 

4 neighbor pixel positions (u,v) by using the captured image brightness value to derive an index for 

5 a lookup table memory storing data representing the inverse of response functions of light- 
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sensing elements having different ones of the plurality of sensitivity levels, and retrieving from 
the lookup table memory a compensated captured image brightness value corresponding to the 
index using lookup table data representing the inverse of the response function of a light-sensing 
element having the sensitivity level corresponding to the one of the nearest neighbor pixel 
positions (u,v), the retrieved compensated brightness value being the brightness value I(u,v) at 
the one of the nearest neighbor pixel positions (u,v). 

37. The method of claim 2, wherein the step of estimating respective off-grid brightness 
values at the off-grid positions of the off-grid position array from the respective captured image 
brightness values at the pixel positions of the pixel position array comprises estimating the 
brightness value at each one of the off-grid positions x\ where x' has values from 2 to xSize - 2, 
xSize being the dimension of the captured line image, the step of estimating the off-grid 
brightness value at each one of the off-grid positions x' comprising: 

(i) comparing a respective brightness value I(k) at each one of four pixel positions 
k that are nearest neighbors to the one of the off- grid positions x' with a predetermined low 
threshold value and a predetermined high threshold value, where k has values from x'- 1 .5 to 
x'+1.5; 

(ii) for each one of the brightness values I(k) at the four nearest neighbor pixel 
positions k that is greater than the predetermined low threshold value and less than the 
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13 predetermined high threshold value computing the quantity I(k) G(k-x'), where G(k-x') is the 

14 value of a 4 x 1 Gaussian interpolation kernel G at position (k-x'); 

15 (iii) adding the quantities I(k) G(k-x') computed in step (ii) to derive a first 

16 sump; 

17 (iv) adding the quantities G(k-x') used in the computations of step (ii) to derive a 

1 8 second sum q; and 

19 (v) computing the estimated brightness value at the one of the off-grid position x' 
r|0 by dividing the first sum p by the second sum q. 

""I 38. The method of claim 37, wherein the respective brightness value I(k) at nearest 

"^2 neighbor pixel position k compared with the predetermined low threshold value and the 

□3 predetermined high threshold value in step (i) is the captured image brightness value at nearest 

rU4 neighbor pixel position k. 

1 39. The method of claim 37, wherein the brightness value I(k) at nearest neighbor pixel 

2 position k compared with the predetermined low threshold value and the predetermined high 

3 threshold value in step (i) is the captured image brightness value at nearest neighbor pixel 

4 position k compensated by the inverse of a light-sensing element having the sensitivity level 

5 corresponding to pixel position k. 
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40. The method of claim 39, wherein the captured image brightness value at each one of 
the nearest neighbor pixel positions k is compensated by the inverse of the response function of a 
light-sensing element having the sensitivity level corresponding to the one of the nearest 
neighbor pixel positions k by using the captured image brightness value at the one of the nearest 
neighbor pixel positions k to an index for a lookup table memory storing data representing the 
inverse of response functions of light-sensing elements having different ones of the plurality of 
sensitivity levels, and retrieving from the lookup table memory a compensated captured image 
brightness value corresponding to the index using lookup table data representing the inverse of 
the response function of a light-sensing element having the sensitivity level corresponding to the 
one of the nearest neighbor pixel positions k, the retrieved compensated captured image 
brightness value being the brightness value I(k) at the one of the nearest neighbor pixel 
position k. 

41. The method of claim 37, wherein the 4 x 1 Gaussian kernel G has the form: 



1.0 


7.7 


7.7 


1.0 



42. The method of claim 10, wherein the step of resampling the estimated off-grid 
brightness values to derive respective resampled on-grid brightness values at the pixel positions 
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3 of the pixel position array comprises deriving the resampled brightness value at each individual 

4 one of the pixel positions by computing the product of respective off-grid brightness values at a 

5 predetermined number of off-grid positions that are nearest neighbors to the individual one of the 

6 pixel positions and cubic interpolation kernel having the same dimensions as the predetermined 

7 number of off-grid positions. 

1 43. The method of claim 5, wherein the step of deriving pixel brightness values of an 

1=32 output image from the estimated off-grid brightness values comprises resampling the estimated 

UT3 off-grid brightness values to derive resampled on-grid brightness values at the pixel positions of 

|: "4 the pixel position array, and wherein each one of the intermediate rows extends medially between 

m 

; *q5 a respective adjacent pair of pixel rows and each one of the intermediate columns extends 

l!36 medially between a respective adjacent pair of pixel columns, and wherein the resampled 

1 ^7 brightness value at each one of the multiplicity of pixel positions is computed by the relation: 

R 3 3 

I{x-2y-2) = £ £ B(x-\.5 + /, 7-I.5 +7)cb , 

1=0 j=o 13 

8 where x has values from 2.5 to xSize-2.5, y has values from 2.5 to ySize -2.5, 1(x-2, y-2) is the 

9 resampled on-grid brightness value at pixel position (x-2, y-2), B(x- 1 .5+i, y- 1.5+j) is the 

1 0 estimated off-grid brightness value at off-grid position (x- 1 .5+i, y- 1 .5+j), i and j are indices of a 

11 4x4 cubic interpolation kernel (J); the indices i and j each have values from 0 to 3, <|> £J is the 
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value of the interpolation kernel <J) at indices i and j, xSize is the dimension of the captured image 
in the direction of the pixel rows, and ySize is the dimension of the captured image in the 
direction of the pixel columns. 

44. The method of claim 33 or 43, wherein the 4 x 4 cubic interpolation kernel (|) has the 

form: 



0.043 


-0.66 


-0.661 


0.043 


-0.661 


10.28 


10.28 


-0.661 


-0.661 


10.28 


10.28 


-0.661 


0.043 


-0.661 


-0.661 


0.043 



i 



45. The method of claim 2 ? wherein the step of deriving pixel brightness values of an 
output image from the estimated off-grid brightness values comprises resampling the estimated 
off-grid brightness values to derive resampled on-grid brightness values at pixel positions of the 
pixel position array, and wherein each off-grid position is located midway between a respective 
adjacent pair of pixel positions, and the resampled on-grid brightness values I G at each one of the 
pixel positions x is computed by the relation: 

J 0 (*-3) = "if *(x+*)i|r(*), 

A: = - 1.5 

where x is the coordinate of the pixel position having values from 3.5 to xSize-3.5, 1 0 (x-3) is the 
resampled on-grid brightness value at pixel position (x-3), B(x+k) is the estimated off-grid 
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brightness value at off-grid position (x+k), k is a position of a 4 x 1 cubic interpolation kernel 
\|/(k) is the value of the cubic interpolation kernel i|r at position k, the position k has values from 
- 1.5 to 1.5, and xSize is the dimension of the captured line image. 

46. The method of claim 45 , wherein the 4 x 1 cubic interpolation kernel i|x has the 

form: 



-0.96 


13.44 


13.44 


-0.96 



47. The method of claim 6, wherein each one of the intermediate rows extends medially 
between a respective adjacent pair of the pixel rows and each one of the intermediate columns 
extends medially between a respective adjacent pair of the pixel columns, and the step of 
estimating respective off-grid brightness values at the off-grid positions of the off-grid position 
array from respective captured image brightness values at the pixel positions of the pixel position 
array comprises estimating the off-grid brightness value at each one of the off-grid positions 
(x',y') s where x' has values from 0.5 to xSize-0.5 and y' has values from 1 to ySize- 1 for off- 
grid positions located at respective intersections of the intermediate rows and the pixel columns, 
x' has values from 1 to xSize- 1 and y' has values from 0.5 to ySize-0.5 for off-grid positions 
located at respective intersections of the pixel rows and the intermediate columns, and xSize and 
ySize are the dimensions of the captured image in the directions of the pixel rows and the pixel 
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12 columns, respectively, the step of estimating the brightness values at each one of the off-grid 

1 3 positions (x' 5 y ')comprising: 

14 (i) for each one of the off-grid positions (x, y') computing coordinates (u, v) using 

1 5 the relations u = x' + 0.5i - 0.5j and v = y' - 1.5 + 0.5i +0.5j, where i and j are indices of a 4 x 4 

16 cubic interpolation kernel 4>, and each one of the indices i and j has values from 0 to 3; 

17 (ii) for each pair of coordinates (u, v) computed in step (i) using indices i and j, 

1 8 determining whether the coordinates (u,v) are those of a pixel position; 

; ff9 (iii) if the coordinates (u, v) computed in step (i) are the coordinates of a pixel 

fiO position, comparing a brightness value I(u,v) at the pixel position (u,v) with a predetermined 

21 low-threshold value and a predetermined high threshold value; 

%2 (iv) if the coordinates (u, v) computed in step (i) using indices i and j are those of 

i33 a pixel position and the brightness value I(u,v) is greater than the predetermined low threshold 

| U4 value and less than the predetermined high threshold value, then compute the quantities: 

^ab 

ab 

25 and (J)^ 2 , where fy m n is the value of a 4 x 4 cubic interpolation kernel (|) at indices m and n; 

26 (v) adding the quantities 

4>.. Kti 9 v) 



ab 
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27 computed in step (iv) to derive a first sum s; 

28 (vi) adding the quantities (J)^ 2 computed in step (iv) to derive a second sum w; and 

29 (vii) computing the estimated brightness value at the one of the off-grid positions 

30 (x, y) by dividing the first sum s by the second sum w. 

1 48. The method of claim 47, wherein the brightness value I(u,v) at each one of the pixel 

2 positions (u,v) compared with the predetermined low threshold value and the predetermined high 
i;53 threshold value in step (iii) is the captured image brightness value at the one of the pixel 

1=^4 positions (u,v). 

Jpl 49. The method of claim 47 5 wherein the brightness value I(u,v) at each one of the pixel 

(132 positions (u,v) compared with the predetermined low threshold value and the predetermined high 

: ^3 threshold value in step (iii) is the captured image brightness value at the one of the pixel 

|=54 positions (u,v) compensated by the inverse of the response function of a light-sensing element 

5 having a sensitivity level corresponding to the one of the pixel positions (u ? v). 

1 50. The method of claim 49, wherein the captured image brightness value at each one of 

2 the pixel positions (u 5 v) is compensated by the inverse of the response function of a light-sensing 

3 element having the sensitivity level corresponding to the one of the pixel positions (u,v) by using 

4 the captured image brightness value I c (u,v) at the one of the pixel positions (u,v) to derive an 
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index for a lookup table memory storing data representing the inverse of response functions of 
light-sensing elements having different ones of the plurality of sensitivity levels, and retrieving 
from the lookup table a compensated captured image brightness value corresponding to the index 
using lookup table data representing the inverse of the response function of a light-sensing 
element having the sensitivity level corresponding to the one of the pixel positions (u,v), the 
retrieved compensated captured image brightness value being the brightness value I(u ? v) at the 
one of the pixel positions (u,v). 

5 1 . The method of claim 47, wherein the step of estimating the off-grid brightness value 
at each one of the off-grid positions (x, y') is first carried out for off-grid positions located at 
respective intersections of intermediate rows and pixel columns, and then carried out for pixel 
positions located at respective intersections of pixel rows and intermediate columns. 

52. The system of claim 47, wherein the 4 X 4 cubic interpolation kernel (f) has the form: 
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Q\ 53. The method of claim 6, wherein the step of deriving pixel brightness values of an 

\ ^2 output image from the estimated off-grid brightness values comprises resampling the estimated 

;:^3 off-grid brightness values to derive respective resampled on-grid brightness values at pixel 

4 positions of the pixel position array, and each one of the intermediate rows extends medially 

5 between a respective adjacent pair of the pixel rows and each one of the intermediate columns 

6 extends medially between a respective adjacent pair of the pixel columns, and wherein the 

7 resampled on-grid brightness value at each one of the pixel positions (x,y) is computed by the 

8 relation: 
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3 3 

/(x-2j>-2) = E E 5(x+0.5/-0.5/ 1 y-1.5+0.5/+0.5y)ct)., 



9 where x has values from 2.5 to xSize -2.5, y has values from 2.5 to ySize -2.5, 1(x-2, y-2) is 

10 the resampled on-grid brightness value at pixel position (x-2, y-2), 

1 1 B(x+0.5i- 0.5j,y- 1 .5+0.5i+0.5j) is the off-grid brightness value at off-grid position 

;4r2 (x+0.5i-j0.5j, y- 1 .5+0.5i+0.5j), i and j are indices of a 4x4 cubic interpolation kernel (J), 

LP3 indices i and j each have values from 0 to 3, (frj is the value of the interpolation kernel (j) at 

|S f4 indices i and j, xSize is the dimension of the captured image in the direction of the pixel rows, 

%5 and ySize is the dimension of the captured image in the direction of the pixel columns. 

Ul 54. The method of claim 5, wherein the step of deriving pixel brightness values of an 

—2 output image from the estimated off-grid brightness values comprises using a Gaussian kernel for 

3 interpolating the estimated off-grid brightness values to derive respective interpolated brightness 

4 values at the pixel positions of the pixel position array. 
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55. The method of claim 54, wherein the Gaussian kernel has the form: 



0.0037 


0.0101 


0.0166 


0.0166 


0.0101 


0.0037 


0.0101 


0.0275 


0.0452 


0.0452 


0.0275 


0.0101 


0.0166 


0.0452 


0.0743 


0.0743 


0.0452 


0.0166 


0.0166 


0.0452 


0.0743 


0.0743 


0.0452 


0.0166 


0.0101 


0.0275 


0.0452 


0.0452 


0.0275 


0.0101 


0.0037 


0.0101 


0.0166 


0.0166 


0.0101 


0.0037 



i 



56. The method of claim 5, wherein the step of deriving pixel brightness values of an 
output image from the estimated off-grid brightness values comprises using a bi-linear 
interpolation filter kernel for interpolation of the off-grid brightness values to derive respective 
interpolated on-grid brightness values at the pixel positions of the pixel position array. 

57. The method of claim 56, wherein the bi-linear interpolation filter kernel has the form: 



0.5 


0.5 


0.5 


0.5 



I 



58. The method of claim 5 or 7, wherein the step of deriving pixel brightness value of 
an output image from the estimated off-grid brightness values comprises resampling the 
estimated off-grid brightness values to derive respective resampled on-grid brightness values at 
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the pixel positions of the pixel position array, and wherein the step of resampling the estimated 
off-grid brightness values comprises shifting the off-grid position array to coincide with the pixel 
position array, whereby the resampled on-grid brightness values at the pixel positions of the pixel 
position array are the respective estimated off-grid brightness values at the coincident off-grid 
positions of the shifted off-grid position array. 

59. A method for obtaining a relatively high dynamic range image of a scene using a 
relatively low dynamic range image sensor exposed to incident light from the scene for capturing 
an image thereof, the image sensor having a multiplicity of light-sensing elements in an array, 
each light-sensing element having a particular one of a plurality of sensitivity levels to incident 
light in accordance with a predetermined sensitivity pattern for the array of light-sensing 
elements and having a respective response function, each light-sensing element being responsive 
to incident light from the scene for producing a captured image brightness value at a 
corresponding one of a multiplicity of pixel positions of a pixel position array, whereby each one 
of the multiplicity of pixel positions corresponds to a particular one of the plurality of sensitivity 
levels of the light-sensing elements, the method comprising the steps of computing respective 
on-grid brightness values at pixel positions of the pixel position array from respective captured 
image brightness values at the pixel positions of the pixel position array, each one of the 
respective on-grid brightness values being computed from a corresponding plurality of the 



NY02:24 1764.1 



- 112- 




FILE NO. A32708-070050.1 125 

14 captured image brightness values, and deriving pixel brightness values of an output image from 

1 5 the on-grid brightness values. 

1 60. The method of claim 59, wherein the array of light-sensing elements of the image 

2 sensor is a linear array for capturing a line image, and the pixel position array is a linear array 

3 having the multiplicity of pixel positions at respective regularly spaced pixel positions in a pixel 

4 row so as to form a linear captured image brightness value array. 

r)E. 

^=ni 61 . The method of claim 59, wherein the array of light-sensing elements of the image 

j= Z2 sensor is a linear array and the pixel position array is a two-dimensional array having pixel 

j\^3 positions at respective intersections of a plurality of regularly spaced pixel rows and a plurality 

i;34 of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and 

rU5 wherein the linear array of light-sensing elements of the image sensor is sequentially exposed to 
successive regularly spaced linear regions of the scene in multiple exposures, each one of the 

7 exposures producing a respective linear captured image brightness value array, the respective 

8 linear captured image brightness value arrays produced by the multiple exposures being 

9 concatenated to form a two-dimensional captured image brightness value array having respective 
10 captured image brightness values at the pixel positions of the pixel position array. 
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1 62. The method of claim 59, wherein the image sensor has a two-dimensional array of 

2 light-sensing elements, and the pixel position array is a two-dimensional array having the 

3 multiplicity of pixel positions at respective intersections of a plurality of regularly spaced pixel 

4 rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the 

5 pixel columns. 

1 63. The method of claim 59, wherein the step of deriving pixel brightness values of an 

?*£ output image from the on-grid brightness values comprises using the respective on-grid 

LfP brightness values at the pixel positions of the pixel position array as the pixel brightness values 

^4 of the output image. 

q1 64. The method of claim 59, wherein the step of deriving pixel brightness values of an 

fU2 output image from the on-grid brightness values comprises compensating the respective on-grid 

;~3 brightness value at each one of the pixel positions of the pixel position array by the inverse of a 

4 combined response function of the light-sensing elements to obtain respective pixel brightness 

5 values of the output image at the pixel positions of the pixel position array, the combined 

6 response function being the sum of weighted response functions of the light-sensing elements 

7 having different ones of the plurality of sensitivity levels, each one of the weighted response 

8 functions being the response function of a light-sensing element having a respective one of the 



NY02:24I764.1 



- 114- 




FILE NO. A32708-070050.1 125 

9 sensitivity levels multiplied by a predetermined weighting factor for the respective one of the 

10 sensitivity levels. 

1 65. The method of claim 64, wherein the step of compensating the on-grid brightness 

2 value at each one of the pixel positions of the pixel position array comprises the steps of deriving 

3 from the on-grid brightness value at each one of the pixel positions a respective index for a 

4 lookup table memory storing data representing the inverse of the combined response function of 
r™5 the light-sensing elements, and retrieving from the lookup table memory a compensated on-grid 
Lu6 brightness value corresponding to the respective index, the compensated on-grid brightness value 
? tj being the pixel brightness value of the output image at the one of the pixel positions. 

□1 66. The method of claim 59, wherein the step of computing respective on-grid brightness 

i"M2 values at pixel positions of the pixel position array comprises computing the respective on-grid 

;: brightness value at each one of the pixel positions by calculating the product of the captured 

4 image brightness values at a predetermined number of pixel positions that are nearest neighbors 

5 to the one of the pixel positions and an on-grid interpolation filter kernel having the same 

6 dimensions as the predetermined number of pixel positions. 

1 67. The method of claim 66, wherein the image sensor has a two-dimensional array of 

2 light-sensing elements, and the pixel array is a two-dimensional array having the multiplicity of 
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3 pixel positions at respective intersections of a plurality of regularly spaced pixel rows and a 

4 plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel 

5 columns, and wherein the on-grid brightness value at each one of the pixel positions (x,y) of the 

6 pixel position array is calculated by the relation 

I(x-2y-2) = E E /(x-2 + i 9 y-2 + j)$ 

qJ where x has values from 2.5 to xSize - 2.5, y has values from 2.5 to ySize - 2.5, 1(x - 2, y - 2) is 

\M the interpolated brightness value at pixel position (x - 2, y - 2), I c (x - 2 + i, y - 2 + j) is the 

captured image brightness value at pixel position (x - 2 + i, y - 2 + j), i and j are indices of a 5 x 5 

|%;0 on-grid interpolation filter kernel 4>, the indices i and j each have values from 0 to 4, <j)y is the 

r|l value of the on-grid interpolation filter kernel (J) at indices i and j, xSize is the dimension of the 

fi2 captured image in the direction of the pixel rows, and ySize is the dimension of the captured 

;^|3 image in the direction of the pixel columns. 
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68. The method of claim 67, wherein the 5x5 on-grid interpolation filter kernel (f) has 
the form: 



0.043 


-0.618 


-1.322 
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0.043 


-0.618 


9.001 


19.238 


9.001 


-0.618 


-1.322 


19.238 
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19.238 


-1.322 


-0.618 


9.001 


19.238 


9.001 


-0.618 


0.043 


-0.618 


-1.322 


-0.618 


0.043 
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69. The method of claim 59, wherein the step of computing respective on-grid brightness 
values at pixel positions of the pixel position array comprises computing a respective on-grid 
brightness value at each individual one of the pixel positions of the pixel position array including 
the steps of : 

(i) comparing a respective compensated captured image brightness value at each one of a 
predetermined number of pixel positions that are nearest neighbors to the individual one of the 
pixel positions with a predetermined low threshold value and a predetermined high threshold 
value; 

(ii) for each one of the nearest neighbor pixel positions having a compensated captured 
image brightness value greater than the predetermined low threshold value and less than the 
predetermined high threshold value, computing the product of the compensated captured image 
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12 brightness value at the one of the nearest neighbor pixel positions and the value of an on-grid 

13 interpolation filter kernel corresponding to the one of the nearest neighbor pixel positions, the 

14 on-grid interpolation filter kernel having the same dimensions as the predetermined number of 

1 5 nearest neighbor pixel positions; 

16 (iii) computing a first sum of all products computed in step (ii); 

17 (iv) computing a second sum of all values of the on-grid interpolation filter kernel used 

1 8 to compute a product in step (ii); and 

rtp (v) computing the on-grid brightness value at the individual one of the pixel positions by 

! JO dividing the first sum by the second sum, wherein the respective compensated captured image 

? 7A brightness value at each one of the nearest neighbor pixel positions is the respective captured 

; "^2 image brightness value at the one of the nearest neighbor pixel positions compensated by the 

l23 inverse of the response function of a light-sensing element having the sensitivity level 

[34 corresponding to the one of the nearest neighbor pixel positions. 

1 70. The method of claim 69, further comprising the steps of using the captured image 

2 brightness value at each one of the pixel positions of the pixel position array to derive a 

3 respective index for a lookup table storing data representing the inverse of response functions of 

4 light-sensing elements having different ones of the plurality of sensitivity levels, and retrieving 

5 from the lookup table a compensated captured image brightness value corresponding to the 

6 respective index from lookup table data representing the inverse of the response function of the 
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7 light-sensing element having the sensitivity level corresponding to the one of the pixel positions, 

8 the retrieved compensated captured image brightness value being the compensated captured 

9 image brightness value at the one of the pixel positions. 

1 71 . A method for obtaining a relatively high dynamic range image of a scene using a 

2 relatively low dynamic range image sensor exposed to incident light from the scene for capturing 

3 an image thereof, the image sensor having a multiplicity of light-sensing elements in a linear 
;;34 array, each light-sensing element having a particular one of a plurality of sensitivity levels to 

U15 incident light in accordance with a predetermined sensitivity pattern for the array of light-sensing 

"~6 elements and having a respective response function, each light-sensing element being responsive 

JH7 to incident light from the scene for producing a captured image brightness value at a 

q8 corresponding one of a multiplicity of pixel position at respective regularly spaced pixel 

\ =9 positions of a linear pixel position array to form a linear captured image brightness value array, 

^;I0 whereby each one of the multiplicity of pixel positions corresponds to a particular one of the 

1 1 plurality of sensitivity levels, the method comprising the steps of: 

12 (i) for each one of the pixel positions x of the pixel position array, where x has 

13 values from 2.5 to xSize-2.5 and xSize is the dimension of the captured image, comparing each 

14 one of brightness values I(k) at five pixel positions k that are nearest neighbors to pixel position 

1 5 x with a predetermined low threshold value and a predetermined high threshold value, where k 

1 6 has values from x- 2 to x+2; 
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17 (ii) for each one of the brightness values I(k) at the five nearest neighbor pixel 

1 8 positions k that is greater than the predetermined low threshold value and less than the 

19 predetermined high threshold value, computing the quantity I(k)G(k-x), where G(k-x) is the 

20 value of a Gaussian interpolation kernel G at position (k-x); 

21 (iii) adding the quantities I(k)G(k-x) computed in step (ii) to derive a first sum p; 

22 (iv) adding the Gaussian interpolation kernel values G(k-x) for all values of k 

23 where I(k) is greater than the predetermined low threshold value and less than the predetermined 
high threshold value to derive a second sum q; and 

^JB (v) computing a pixel brightness value of an output image I 0 (x) at pixel position x 

] i6 by dividing the first sum p by the second sum q. 

qI 72. The method of claim 71 , wherein the brightness value I(k) at pixel position k 

TU2 compared with the predetermined low threshold value and the predetermined high threshold 

;:!?3 value in step (i) is the captured image brightness value at pixel position k. 

1 73. The method of claim 71, wherein the brightness value I(k) at pixel position k 

2 compared with the predetermined low threshold value and the predetermined high threshold 

3 value in step (i) is the captured image brightness value at pixel position k compensated by the 

4 inverse of response function of a light-sensing element having a sensitivity level corresponding 

5 to pixel position k. 
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1 74. The method of claim 73, wherein the captured image brightness value at pixel 

2 position k is compensated by the inverse of the response function of a light-sensing element 

3 having a sensitivity level corresponding to pixel position k by using the captured image 

4 brightness value at pixel position k to derive a respective index for a lookup table memory 

5 storing data representing the inverse of response functions of light-sensing elements having 

6 different ones of the plurality of sensitivity levels, and retrieving from the lookup table memory a 
r^I compensated captured image brightness value I(k) corresponding to the respective index using 
Ln8 lookup table data representing the inverse of the response function of a light-sensing element 

1 s ^9 having the sensitivity level corresponding to pixel position k. 

rjl 75. The method of claim 72, wherein the 5 x 1 Gaussian kernel G has the form: 



0.1 


3.6 


10.0 


3.6 


0.1 



□4 k 



1 76. The method of claim 31, 35, 39, 49, 69 or 73, wherein the predetermined low 

2 threshold value is the noise level of a light-sensing element having the sensitivity level 

3 corresponding to the pixel position having the compensated brightness value being compared 

4 therewith compensated by the inverse of the response function of a light-sensing element having 
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5 the corresponding sensitivity level, and the predetermined high threshold value is the saturation 

6 or near saturation brightness value of a light-sensing element having the same sensitivity level. 

1 77. A system for obtaining capturing a relatively high dynamic range image of a scene 

2 using a relatively low dynamic range image sensor adapted to be exposed to incident light from 

3 the scene for capturing an image thereof comprising: 

4 (a) an image sensor having a multiplicity of light-sensing elements in an array, each light- 
q5 sensing element having a particular one of a plurality of sensitivity levels to incident light in 

|=H6 accordance with a predetermined sensitivity pattern for the array of light-sensing elements and a 

ls Z7 respective response function, each light-sensing element being responsive to incident light from 

; g8 the scene for producing a captured image brightness value at a corresponding one of a 

i:39 multiplicity of pixel positions of a pixel position array, whereby each one of the multiplicity of 

; WO pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light- 

;;i 1 sensing elements; 

12 (b) a captured image memory for storing the captured image brightness values produced 

13 by the light-sensing elements at corresponding ones of the multiplicity of pixel positions of the 

14 pixel position array; 

15 (c) an off-grid estimator for deriving from the captured image brightness values in the 

16 captured image memory respective estimated off-grid brightness values at a multiplicity of off- 
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1 7 grid positions located at respective interstices of the pixel position array so as to form a regular 

1 8 off-grid position array; and 

19 (d) an output image generator for deriving pixel brightness values of an output image 

20 from the estimated off-grid brightness values. 

1 78. The system of claim 77, wherein the array of light-sensing elements of the image 

2 sensor is a linear array for capturing a line image and the pixel position array is a linear array 
[33 having the multiplicity of pixel positions located at respective regularly spaced positions in a 
104 pixel row so as to form a linear captured image brightness value array, the captured image 

j= I5 memory for storing the captured image brightness values produced by the light-sensing elements. 

□1 79. The system of claim 78, wherein the off-grid positions of the off-grid position array 

i-U2 are located between respective adjacent pairs of pixel positions of the linear pixel position array. 

1 80. The system of claim 79, wherein the predetermined sensitivity pattern has a 

2 predetermined first sensitivity level and a predetermined second sensitivity level, and adjacent 

3 pixel positions of the linear pixel position array correspond to different ones of the predetermined 

4 first sensitivity level and the predetermined second sensitivity level. 
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1 81. The system of claim 77, wherein the array of light-sensing elements of the image 

2 sensor is a linear array and the pixel position array is a two-dimensional array having pixel 

3 positions at respective intersections of a plurality of regularly spaced pixel rows and a plurality 

4 of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and 

5 wherein the linear array of light-sensing elements of the image sensor is adapted to be 

6 sequentially exposed to successive regularly spaced linear regions of the scene in multiple 

7 exposures, each one of the exposures producing a respective linear captured image brightness 
?=8 value array, the respective linear captured image brightness value arrays produced by the 

j s n9 multiple exposures being concatenated to form a two-dimensional captured image brightness 

: = f0 value array having respective captured image brightness values at the pixel positions of the pixel 

; s j;l position array, the concatenated linear captured image brightness value arrays being stored in the 

1+2 captured image memory as respective captured image brightness values at the pixel positions of 

fU3 the two-dimensional pixel position array. 

1 82. The system of claim 81, wherein the off-grid positions of the off-grid position array 

2 are located at respective intersections of a plurality of regularly spaced intermediate columns and 

3 pixel rows, each one of the intermediate columns being parallel to the pixel columns and 

4 extending between a respective adjacent pair of the pixel columns. 
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1 83. The system of claim 82, wherein the predetermined sensitivity pattern has a first 

2 predetermined sensitivity level and a second predetermined sensitivity level, and wherein 

3 adjacent pixel positions in each one of the pixel rows correspond to different ones of the first and 

4 second predetermined sensitivity levels, and adjacent pixel positions in each one of the pixel 

5 columns correspond to the same one of the first and second predetermined sensitivity levels. 

1 84. The system of claim 77, wherein the image sensor has a two-dimensional array of 
q2 light-sensing elements and the multiplicity of pixel positions are located at respective 

Lm3 intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel 

^4 columns, the pixel rows being orthogonal to the pixel columns, the captured image memory for 

; s s5 storing the captured image brightness values produced by the light-sensing elements at 

[□6 corresponding ones of the multiplicity of pixel positions of the pixel position array. 

~1 85. The system of claim 84 wherein the off-grid positions of the off-grid position array 

2 are located at respective intersections of a plurality of regularly spaced intermediate rows and a 

3 plurality of regularly spaced intermediate columns, each intermediate row being parallel to the 

4 pixel rows and extending between a respective adjacent pair of pixel rows and each intermediate 

5 column being parallel to the pixel columns and extending between a respective adjacent pair of 

6 pixel columns. 
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1 86. The system of claim 85, wherein the predetermined sensitivity pattern of the array of 

2 light-sensing elements has four different predetermined sensitivity levels and corresponds to a 

3 repetitive pattern of groups of four nearest neighbor pixel positions, each one of the four pixel 

4 positions in each group corresponding to a different one of the four predetermined sensitivity 

5 levels in a common predetermined positional order. 

1 87. The system of claim 84, wherein the off-grid positions of the off-grid position array 

HI are located at respective intersections of a plurality of regularly spaced intermediate rows and the 

Lfj3 plurality of pixel columns, and at respective intersections of a plurality of regularly spaced 

|: f4 intermediate columns and the pixel rows, each one of the intermediate rows being parallel to the 

/^5 pixel rows and extending between a respective adjacent pair of the pixel rows, each one of the 

q6 intermediate columns being parallel to the pixel columns and extending between a respective 

fU7 adjacent pair of pixel columns. 

1 88. The system of claim 87, wherein the predetermined sensitivity pattern of the array of 

2 light-sensing elements has first and second predetermined sensitivity levels, and wherein 

3 adjacent pixel positions in each one of the pixel rows and adjacent pixel positions in each one of 

4 the pixel columns correspond to different ones of the first and second predetermined sensitivity 

5 levels. 
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1 89. The system of claim 77, wherein the output image generator comprises an output 

2 image memory for storing the respective estimated off-grid brightness values derived by the off- 

3 grid estimator as the pixel brightness values of the output image. 

1 90. The system of claim 77, wherein the output image generator comprises: 

2 (a) an off-grid brightness value memory for storing the estimated off-grid brightness 

3 values derived by the off-grid estimator; 

q4 (b) an on-grid resampler for deriving from the estimated off-grid brightness values in the 

Ln5 off-grid brightness value memory respective resampled on-grid brightness values at the pixel 

positions of the pixel position array; and 
^Cl (c) an output image memory for storing the respective resampled on-grid brightness 

i;38 values derived by the on-grid resampler as the pixel brightness values of the output image. 

;:S;1 91. The system of claim 77, wherein the off-grid estimator comprises: 

2 (a) an off-grid position generator for providing the off-grid positions of the off-grid 

3 position array; 

4 (b) a sensitivity pattern memory for storing data indicative of the sensitivity level 

5 corresponding to each one of the pixel positions of the pixel position array; 

6 (c) a plurality of sensitivity level off-grid brightness value estimators each corresponding 

7 to a respective one of the plurality of sensitivity levels of the light-sensing elements, and 
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8 receiving off-grid positions from the off-grid position generator and sensitivity pattern data from 

9 the sensitivity pattern memory, each one of the sensitivity level off-grid brightness value 

1 0 estimators being responsive to an off-grid position received from the off-grid position generator 

1 1 and the sensitivity pattern data from the sensitivity pattern memory for deriving from the 

12 captured image brightness values in the captured image memory a respective sensitivity level 

13 off-grid brightness value for the corresponding sensitivity level at the received off-grid position; 

14 and 

i+5 (d) an accumulator for combining the respective sensitivity level off-grid brightness 

"z'zzi 

Lfl6 values derived by the plurality of sensitivity level off-grid brightness value estimators for the 

|: f7 corresponding sensitivity levels at each off-grid position received from the off-grid position 

^8 generator to derive a respective estimated off-grid brightness value at each off-grid position 

i|9 received from the off-grid position generator. 

ru 

92. The system of claim 90, wherein the on-grid resampler comprises: 

2 (i) a pixel position generator for providing the pixel positions of the pixel 

3 position array; 

4 (ii) an interpolator for receiving pixel positions provided by the pixel position 

5 generator and being responsive to a pixel position received from the pixel position generator for 

6 deriving from the estimated off-grid brightness values in the off-grid brightness value memory a 

7 respective interpolated on-grid brightness value at the received pixel position; and 
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8 (iii) a response function compensator for compensating the respective interpolated 

9 on-grid brightness value derived by the interpolator at each pixel position received from the pixel 

10 position generator by the inverse of a combined response function of the light-sensing elements 

1 1 to derive a respective resampled on-grid brightness value at each pixel position received from the 

12 pixel position generator, the combined response function of the light-sensing elements being the 

13 sum of weighted response functions of light-sensing elements having different ones of the 

14 sensitivity levels, each one of the weighted response functions being a response function of a 
i;|5 light-sensing element having a respective one of the sensitivity levels multiplied by a 

106 predetermined weighting factor for the respective one of the sensitivity levels. 

Jgl 93. The system of claim 92, wherein the response function compensator comprises a 

□2 lookup table memory storing data representing the inverse of the combined response function of 

i M3 the light-sensing elements and a mapper for deriving from the respective interpolated on-grid 

^4 brightness value derived by the interpolator at a received pixel position a respective index for the 

5 lookup table memory and providing a data value in the lookup table memory corresponding to 

6 the respective index to the output image memory, the data value being the pixel brightness value 

7 of the output image at the received pixel position. 

1 94. The system of claim 91, wherein the accumulator comprises an adder for adding the 

2 sensitivity level off-grid brightness values derived by the plurality of sensitivity level off-grid 
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3 brightness value estimator for corresponding sensitivity levels at each received off-grid position 

4 to derive the respective estimated off-grid brightness value at each off-grid positions received 

5 from the off-grid position generator. 

1 95. The system of claim 91, wherein the accumulator comprises a weighting factor 

2 memory for storing a plurality of predetermined weighting factors each corresponding to a 

3 different one of the plurality of sensitivity levels of the light-sensing elements, a plurality of 
1=^4 multipliers each for multiplying the sensitivity level off-grid brightness value corresponding to a 
Lf|5 respective one of the sensitivity levels at a received off-grid position by the weighting factor 
^6 corresponding to the respective one of the sensitivity levels to provide a respective weighted 
| = L{7 sensitivity level off-grid brightness value for the corresponding sensitivity level at the received 
□8 off-grid position, and an adder for summing the respective weighted sensitivity level off-grid 
!'U9 brightness values provided by the plurality of multipliers to derive a respective estimated off-grid 
i;|0 brightness value at the received off-grid position received from the off-grid position generator. 

1 96. The system of claim 77, wherein the off-grid estimator comprises: 

2 (a) an off-grid position generator for providing the off-grid positions of the off-grid 

3 position array; 

4 (b) a sensitivity pattern memory for storing data indicative of the sensitivity level 

5 corresponding to each one of the pixel positions of the pixel position array; 
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6 (c) a plurality of sensitivity level off-grid brightness value estimators, each 

7 corresponding to a respective one of the sensitivity levels, for receiving off-grid positions from 

8 the off-grid position generator and sensitivity pattern data from the sensitivity pattern memory, 

9 each one of the plurality of sensitivity level off-grid brightness value estimators being responsive 

10 to an off-grid position received from the off-grid position generator and the sensitivity pattern 

1 1 data from the sensitivity pattern memory for deriving from the captured image brightness values 

12 in the captured image memory a respective sensitivity level off-grid brightness value for the 
i+3 corresponding sensitivity level at the received off-grid position; 

LE4 (d) an accumulator for combining the respective sensitivity level off-grid brightness 

l= j5 values derived by the plurality of sensitivity level off-grid brightness value estimator for the 

"3j6 corresponding sensitivity levels at each off-grid position received from the off-grid position 

[}7 generator to derive a respective combined sensitivity level off-grid brightness value at each off- 

!*W8 grid position received from the off-grid position generator; and 

;;i9 (e) a response function compensator for compensating the respective combined sensitivity 

20 level off-grid brightness value at each off-grid position received from the off-grid position 

21 generator by the inverse of a combined response function of the light-sensing elements to derive 

22 a respective estimated off-grid brightness value at each off-grid position received from the off- 

23 grid position generator, the combined response function being the sum of weighted response 

24 functions of light-sensing elements having different ones of the plurality of sensitivity levels, 

25 each one of the weighted response functions being a response function of a light-sensing element 
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26 having a respective one of the sensitivity levels multiplied by a predetermined weighting factor 

27 for the respective one of the sensitivity levels. 

1 97. The system of claim 96, wherein the response function compensator comprises a 

2 lookup table memory storing lookup table data representing the inverse of the combined response 

3 function of the light-sensing elements, and a mapper for deriving from the respective combined 

4 sensitivity level off-grid brightness value at each off-grid position received from the off-grid 
□5 position generator a respective index for the lookup table memory and providing a data value in 
Lm6 the lookup table memory corresponding to the respective index as the respective estimated off- 
i= l7 grid brightness value at the off-grid position received from the off-grid position generator. 

Ql 98. The system of claim 96, wherein the output image generator comprises: 

rU2 (a) an off-grid brightness value memory for storing the respective estimated off-grid 

!:^3 brightness value derived by the response function compensator at each off-grid position received 

4 from the off-grid position generator; 

5 (b) an on-grid resampler for deriving from the estimated off-grid brightness values in the 

6 off-grid brightness value memory respective resampled on-grid brightness values at the pixel 

7 positions of the pixel position array; and 

8 (c) an output image memory for storing the respective resampled on-grid brightness 

9 values derived by the on-grid resampler as the pixel brightness values of the output image, 



NY02:241764.1 



- 132- 




FILE NO. A32708-070050.1 125 

10 and wherein the on-grid resampler comprises: 

11 (i) a pixel position generator for providing pixel positions of the pixel position 

12 array; and 

13 (ii) an interpolator for receiving pixel positions provided by the pixel position 

14 generator and being responsive to a pixel position received from the pixel position generator for 

15 deriving from the estimated off-grid brightness values in the off-grid brightness value memory a 

16 respective resampled on-grid brightness value at the received pixel position. 

Lfil 99. The system of claim 96, wherein the accumulator of the off-grid estimator comprises 

i= ;j2 an adder for adding the respective sensitivity level off-grid brightness values derived by the 

= p3 plurality of sensitivity level off-grid brightness value estimators for the corresponding sensitivity 

i;:j4 levels at each off-grid position received from the off-grid position generator to provide the 

;"U5 combined sensitivity level off-grid brightness values at each off-grid position received from the 

;;^6 off-grid position generator. 

1 100. The system of claim 96, wherein the accumulator of the off-grid estimator 

2 comprises a weighting factor memory for storing a plurality of predetermined weighting factors 

3 each corresponding to a different one of the plurality of sensitivity levels of the light-sensing 

4 elements, a plurality of multipliers each for multiplying the sensitivity level off-grid brightness 

5 value corresponding to a respective one of the sensitivity levels at a received off-grid position by 
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6 the weighting factor corresponding to the respective one of the sensitivity levels to provide a 

7 weighted sensitivity level off-grid brightness value for the corresponding sensitivity level at the 

8 off-grid position received from the off-grid position generator, and an adder for summing the 

9 weighted sensitivity level off-grid brightness values at the received off-grid position provided by 

10 the plurality of multipliers to derive the combined sensitivity level off-grid brightness value at 

1 1 the off-grid positions received from the off-grid position generator. 

qI 101 . The system of claim 100, wherein the array of light-sensing elements of the image 

■ j =R2 sensor is a linear array and the pixel position array is a linear array having a multiplicity of pixel 

|s 3 positions at respective regularly spaced pixel positions in a pixel row, and wherein the off-grid 

^4 positions of the off-grid position array are each located between a respective adjacent pair of 

q5 pixel positions of the pixel position array and the sensitivity level off-grid brightness values 

fU6 derived by the plurality of sensitivity level off-grid brightness value estimators for an off-grid 

:^7 position received from the off-grid position generator are respective captured image brightness 

8 values at a predefined number of pixel positions that are nearest neighbors to the received off- 

9 grid position, each one of the predefined number of nearest neighbor pixel positions 

10 corresponding to a different one of the plurality of sensitivity levels of the light-sensing 

1 1 elements. 
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1 1 02. The system of claim 101, wherein each one of the off-grid positions of the off-grid 

2 position array is located midway between a respective adjacent pair of pixel positions of the 

3 linear pixel position array and the predetermined sensitivity pattern of the light-sensing elements 

4 has a first and a second predetermined sensitivity level, and wherein the combined sensitivity 

5 level off-grid brightness value at an off-grid position x' received from the off-grid position 

6 generator is expressed as 

7 B(x') = W x I c (x' - 0.5) + W 2 I c (x' + 0.5), 

138 where B(x') is the combined sensitivity level off-grid brightness value at the received off-grid 

Ln9 position x', W! is the weighting factor for the sensitivity level corresponding to pixel position 

l= f0 (x' - 0.5), W 2 is the weighting factor for the sensitivity level corresponding to pixel position 
(x' + 0.5), I c (x' - 0.5) is the captured image brightness value at pixel position (x' - 0.5) and 

i:h2 I c (x' + 0.5) is the captured image brightness value at pixel position (x' + 0.5), and the combined 

rH3 response function of the light-sensing elements is expressed as 
| 4 S(E) = W { P { (E) + W 2 P 2 (E), 

1 5 where P^E) is the radiometric response function of a light-sensing element having the sensitivity 

16 level corresponding to pixel position (x' - 0.5) and P 2 (E) is the radiometric response function of 

17 a light-sensing element having the sensitivity level corresponding to pixel position (x' + 0.5). 

1 103. The system of claim 100, wherein the image sensor has a two-dimensional array of 

2 light-sensing elements and the multiplicity of pixel positions are located at respective 
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3 intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel 

4 columns defining a two-dimensional pixel position array, the pixel rows being orthogonal to the 

5 pixel columns, the multiplicity of off-grid positions are located at respective intersections of a 

6 plurality of regularly spaced intermediate rows and a plurality of regularly spaced intermediate 

7 columns, each one of the intermediate rows being parallel to the pixel rows and extending 

8 between a respective adjacent pair of the pixel rows, each one of the intermediate columns being 

9 parallel to the pixel columns and extending between a respective adjacent pair of the pixel 
i]|0 columns. 

I= ^l 104. The system of claim 103, wherein the predetermined sensitivity pattern of the array 

:"q2 of light-sensing elements corresponds to a pixel position array having repetitive disposed groups 

□3 of four nearest neighbor pixel positions, where each one of the four nearest neighbor pixel 

; 44 positions in each group corresponds to a respective one of four predetermined sensitivity levels 

i;S5 in a common predetermined positional order. 

1 105. The system of claim 100, wherein the image sensor has a two-dimensional array of 

2 light-sensing elements and the multiplicity of pixel positions are located at respective 

3 intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel 

4 columns defining a two-dimensional pixel position array, the pixel rows being orthogonal to the 

5 pixel columns, and wherein the multiplicity of off-grid positions are located at respective 
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6 intersections of a plurality of regularly spaced intermediate rows and the pixel columns, and at 

7 respective intersections of a plurality of regularly spaced intermediate columns and the pixel 

8 rows, each one of the intermediate rows being parallel to the pixel rows and extending between a 

9 respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to 
10 the pixel columns and extending between a respective adjacent pair of the pixel columns. 

1 106. The system of claim 100, wherein the predetermined sensitivity pattern of the array 

i-~2 of light-sensing elements has a first and a second predetermined sensitivity level, and adjacent 

LFB pixel positions in each one of the pixel rows and adjacent pixel positions in each one of the pixel 

|S "4 columns correspond to different ones of the predetermined first and second sensitivity levels. 

pi 107. The system of claim 100, wherein the array of light-sensing elements of the image 

fU2 sensor is a linear array and the multiplicity of pixel positions are located at respective regularly 

;:^3 spaced pixel positions in a pixel row so as to form a linear captured image brightness value array, 

4 and wherein the linear array of light-sensing elements of the image sensor is adapted to be 

5 sequentially exposed to successive regularly spaced linear regions of the scene in multiple 

6 exposures, each one of the exposures producing a respective linear captured image brightness 

7 value array, the respective linear captured image brightness value arrays produced by the 

8 multiple exposures being concatenated to form a two-dimensional captured image brightness 

9 value array having respective captured image brightness values at pixel positions of a pixel 
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10 position array located at respective intersections of a plurality of regularly spaced pixel rows and 

1 1 a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel 

12 columns, and wherein the multiplicity of off-grid positions are located at respective intersections 

13 of the pixel rows and a plurality of regularly spaced intermediate columns, each one of the 

14 intermediate columns being parallel to the pixel columns and extending between a respective 

1 5 adjacent pair of the pixel columns. 

pi 108. The system of claim 107, wherein the predetermined sensitivity pattern of the array 

i=n2 of light-sensing elements has a first and a second predetermined sensitivity level, and wherein 

3 adjacent pixel positions in each pixel row correspond to different ones of the first and the second 

"^4 predetermined sensitivity levels and adjacent pixel positions in each pixel column correspond to 

q5 the same one of the first and the second predetermined sensitivity levels. 

1 109. The apparatus of claim 91 or 96, wherein each one of the sensitivity level off-grid 

2 brightness value estimators is responsive to an off-grid position received from the off-grid 

3 position generator and the data indicative of the sensitivity level corresponding to each one of the 

4 pixel positions from the sensitivity pattern memory for determining a pixel position 

5 corresponding to the same sensitivity level as the sensitivity level off-grid brightness value 

6 estimator and located nearest the received off-grid position, and for estimating a sensitivity level 

7 off-grid brightness value at the received off-grid position to be equal to the captured image 
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8 brightness value at the nearest pixel position corresponding to the same sensitivity level as the 

9 sensitivity level off-grid brightness value estimator. 

1 110. The method of claim 92, wherein the interpolator is responsive to a pixel position 

2 received from the pixel position generator for computing a product of the estimated off-grid 

3 brightness values at a predetermined number of off-grid positions that are nearest neighbors to 

4 the received pixel position and an interpolation kernel having the same dimensions as the 
r^S predetermined number of off-grid positions. 

' ; fl 111. The apparatus of claim 92, wherein the pixel positions provided by the pixel 

^2 position generator are located at respective intersections of a plurality of regularly spaced pixel 

□3 rows and a plurality of regularly spaced pixel columns, the pixel columns being orthogonal to the 

f U4 pixel rows, and the off-grid positions provided by the off-grid position generator are located at 

L ;f 5 respective intersections of a plurality of regularly spaced intermediate rows and a plurality of 

6 regularly spaced intermediate columns, each one of the intermediate rows being parallel to the 

7 pixel rows and extending medially between a respective adjacent pair of the pixel rows, each 

8 intermediate column being parallel to the pixel columns and extending medially between a 

9 respective adjacent pair of the pixel columns, and wherein the interpolator derives the 

10 interpolated on-grid brightness values at each one of the pixel positions (x,y) received from the 

1 1 pixel position generator using the relation 
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I(x-2y-2) - E £ B(x-l.S + z, jM.5 + 7)<b, 

where x has values from 2.5 to xSize -2.5, y has values from 2.5 to ySize -2.5, 1(x-2, y-2) is 
the interpolated on-grid brightness value at pixel position (x-2, y-2), B(x- 1.5+i, y- 1.5+j) is the 
off-grid brightness value at off-grid position (x- 1.5+i, y- 1.5+j), (foj is the value of a 4 x 4 cubic 
interpolation kernel <$> at indices i and j, indices i and j each has values from 0 to 3, xSize is the 
dimension of the captured image in the direction of the pixel rows, and ySize is the dimension of 
the captured image in the direction of the pixel columns. 

112. The apparatus of claim 98, wherein the pixel positions provided by the pixel 
position generator are located at respective intersections of a plurality of regularly spaced pixel 
rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the 
pixel columns, and the off-grid positions provided by the off-grid position generator are located 
at respective intersections of a plurality of regularly spaced intermediate rows and a plurality of 
regularly spaced intermediate columns, each one of the intermediate rows being parallel to the 
pixel rows and extending medially between a respective adjacent pair of the pixel rows, each one 
of the intermediate columns being parallel to the pixel columns and extending medially between 
a respective adjacent pair of the pixel columns, and wherein the interpolator derives the 
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10 resampled on-grid brightness values at each one of the pixel positions (x 5 y) received from the 

1 1 pixel position generator by the relation 

Kx-2y-2) - E E B(x-1.5 + i 9 y-1.5 +7)*, 

i=o j=0 iJ 

12 where x has values from 2.5 to xSize-2.5, y has values from 2.5 to ySize-2.5, I(x-2, y-2) is the 

13 resampled on-grid brightness value at pixel position (x-2, y-2), B(x- 1.5+i,y- 1.5+j) is the off- 
i;|4 grid brightness value at off-grid position (x- 1 .5+i, y- 1 .5+j), (j^ is the value of a 4 x 4 cubic 
LB5 interpolation kernel cj> at indices i and j, the indices i and j each have values from 0 to 3, xSize is 
|S f 6 the dimension of the captured image in the direction of the pixel rows, and ySize is the 

"%7 dimension of the captured image in the direction of the pixel columns. 

!"U1 113. The apparatus of claim 92, wherein the pixel positions provided by the pixel 

position generator are located at respective intersections of a plurality of regularly spaced pixel 

3 columns and a pluraity of regularly spaced pixel rows, the pixel rows being orthogonal to the 

4 pixel columns, and the off-grid positions provided by the off-grid generator are located at 

5 respective intersections of a plurality of regularly spaced intermediate rows and the pixel 

6 columns and at respective intersections of a plurality of regularly spaced intermediate columns 

7 and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and 

8 extending medially between a respective adjacent pair of the pixel rows, each one of the 
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9 intermediate columns being parallel to the pixel columns and extending medially between a 

10 respective adjacent pair of the pixel columns, and wherein the interpolator derives the 

1 1 interpolated on-grid brightness values at each one of the pixel positions (x,y) received from the 

12 pixel position generator using the relation 

3 3 

/(x-2j^2) = E E *(x+0.5i-0.5/,y-1.5+0.5^^^ 

i=o J=Q J 

r|3 where x has values from 2.5 to xSize -2.5, y has values from 2.5 to ySize -2.5, 1(x-2, y-2) is 

LH4 the interpolated on-grid brightness value at pixel position (x-2, y-2), 

jS f 5 B(x+0.5i-0.5j, y- 1.5+0. 5i+0.5j) is the off-grid brightness value at off-grid position 

'%6 (x+0.5i-0.5j, y- 1 .5+0.5i+0.5j), (frj is the value of a 4 x 4 cubic interpolation kernel (J) at indices i 

;;|7 and j, indices i and j each have values from 0 to 3, xSize is the dimension of the captured image 

rl8 in the direction of the pixel rows, and ySize is the dimension of the captured image in the 

;:f9 direction of the pixel columns. 

1 114. The apparatus of claim 98, wherein the pixel positions provided by the pixel 

2 position generator are located at respective intersections of a plurality of regularly spaced pixel 

3 rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the 

4 pixel columns, and the off-grid positions provided by the off-grid position generator are located 

5 at respective intersections of a plurality of regularly spaced intermediate rows and the pixel 
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6 columns and at respective intersections of a plurality of regularly spaced intermediate columns 

7 and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and 

8 extending medially between a respective adjacent pair of the pixel rows, each one of the 

9 intermediate columns being parallel to the pixel columns and extending medially between a 

10 respective adjacent pair of the pixel columns, and wherein the interpolator derives the resampled 

1 1 on-grid brightness values at each one of the pixel positions (x,y) received from the pixel position 

1 2 generator using the relation 

□ 3 3 

,0 I(x-2y-2) = Y, E 5(x+0.5i-0.5y", y-1.5+0.5/+0.5/)4>.* 

Ln i=° y=o lJ 

/to where x has values from 2.5 to xSize-2.5, y has values from 2.5 to ySize-2.5, 1(x-2,y-2) is the 

i;|4 resampled on-grid brightness value at pixel position (x-2,y-2), B(x+0.5i-0.5j, y- 1 .5+0.5i+0.5j) 

l 8 H5 is the off-grid brightness value at the off-grid position (x+0.5i-0.5j, y- 1 .5+0.5i+0.5j), <j>y is the 

:?6 value of a 4 x 4 cubic interpolation kernel (J> at indices i and j, indices i and j each have values 

1 7 from 0 to 3, xSize is the dimension of the captured image in the direction of the pixel rows, and 

1 8 ySize is dimension of the captured image in the direction of the pixel columns. 
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115. The method of claim 1 1 1 or 1 12, wherein the 4 x 4 cubic interpolation kernel (|) has 
the form: 
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116. The system of claim 98, wherein the pixel positions provided by the pixel position 
generator are located at respective regularly spaced pixel positions in a pixel row defining a 
linear pixel position array and the off-grid positions provided by the off-grid position generator 
are each located midway between a respective adjacent pair of pixel positions of the pixel 
position row, and wherein the interpolator derives the resampled on-grid brightness values at 
each one of the pixel positions x received from the pixel position generator using the relation 

- 3 > = X? b(x + mk\ 

k=-i.5 

where x has values from 3.5 to xSize-3.5, 1 0 (x - 3) is the resampled on-grid brightness value at 
pixel position (x - 3), B(x + k) is the estimated off-grid brightness value at off-grid position 
(x + k), k is a position of a 4 x 1 cubic interpolation kernel i|r(k) is the value of the cubic 
interpolation kernel i|j at position k, the position k has values from - 1 .5 to 1 .5, and xSize is the 
dimension of the captured line image. 
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1 117. The system of claim 116, wherein the 4 * 1 cubic interpolation kernel x has the 

2 form: 



-0.96 


13.44 


13.44 


-0.96 



4 

5 k 

1 118. The apparatus of claim 77, wherein the image sensor is an integrated circuit charge 

2 coupled device having an array of photodiodes in spaced orthogonal rows and columns, each one 
es= 3 of the photodiodes having a light-sensing surface adapted to be exposed to incident light and 
f^4 being operatively biased to store photogenerated charge therein, the charge coupled device 

M5 having respective integrated lenses formed over the photosensitive surfaces of predetermined 

ones of the photodiodes, each integrated lens concentrating the incident light onto the respective 

*J7 light-sensing surface, whereby the sensitivity level of each one of the photodiodes is determined 

rH8 by whether or not an integrated lens is formed over the light-sensing surface thereof. 

1 119. The apparatus of claim 77, wherein the image sensor is an integrated circuit charge 

2 coupled device having an array of photodiodes arranged in spaced orthogonal rows and columns, 

3 each one of the photodiodes having a light-sensing surface adapted to be exposed to incident 

4 light and being operatively biased to store photogenerated charge therein, each one of the 

5 photodiodes having an integrated light filter formed above the light-sensing surface thereof, the 

6 transparency of each integrated light filter being fixed during fabrication of the charge coupled 
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7 device, whereby the sensitivity level of each one of the photodiodes in the array is determined by 

8 the transparency of the integrated light filter above the light-sensing surface thereof. 

1 120. The system of claim 77, wherein the image sensor is an integrated circuit charge 

2 coupled device having an array of photodiodes arranged in spaced orthogonal rows and columns, 

3 each one of the photodiodes having a light-sensing surface adapted to be exposed to incident 

4 light and being operatively biased to store photogenerated charge therein, the charge coupled 
35 device being formed with an opaque layer overlying the array of photodiodes with a respective 
H6 aperture formed in the opaque layer above the light-sensing surface of each one of the 

17 photodiodes, the respective size of the aperture above the light-sensing surface of each one of the 

^8 photodiodes being fixed during fabrication of the charge coupled device, whereby the sensitivity 

□9 level of each one of the photodiodes in the array is determined by the size of the aperture above 

U0 the light-sensing surface thereof. 

1 121 . The system of claim 77, wherein the image sensor is an integrated circuit charge 

2 coupled device having a multiplicity of photodiodes in an array of spaced orthogonal rows and 

3 columns, each one of the photodiodes having a light-sensing surface adapted to be exposed to 

4 incident light and being operatively biased to store photogenerated charge therein, the charge 

5 coupled device having an interline structure in which the columns of photodiodes are each 

6 disposed adjacent a corresponding vertical shift register, each photodiode having an associated 
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7 transfer gate electrode for controlling the transfer of stored photogenerated charge from the 

8 photodiode to the corresponding vertical shift register, the stored photogenerated charge in each 

9 one of the photodiodes being operatively periodically removed by the application of a periodic 

10 reset pulse to the device, the photogenerated charge stored in each one of the photodiodes being 

1 1 operatively transferred to a respective stage of the corresponding vertical shift register by the 

12 application of a transfer trigger pulse to the associated transfer gate electrode and being 

13 accumulated therein for an exposure interval, the transfer trigger pulse being applied immediately 
r|4 preceding a reset pulse, the respective transfer trigger pulses being operatively applied to the 

115 transfer gate electrodes associated with the photodiodes in the array at predetermined 

i= j6 frequencies, whereby the sensitivity level of each one of the photodiodes in the array is 

"•£7 determined by the frequency of transfer trigger pulses applied to the transfer gate electrode 

;;J8 associated therewith. 

1 122. The system of claim 77, wherein the off-grid estimator comprises: 

2 (a) a pixel position generator for providing the pixel positions of the pixel position array; 

3 (b) a sensitivity pattern memory for storing the sensitivity level corresponding to each 

4 one of the pixel positions; 

5 (c) a response function compensator receiving pixel positions of the pixel position array 

6 from the pixel position generator, data indicative of the sensitivity levels corresponding to the 

7 received pixel positions from the sensitivity pattern memory and captured image brightness 
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8 values at the received pixel positions from the captured image memory, and being responsive to a 

9 pixel position received from the pixel position generator and data indicative of the sensitivity 

10 level corresponding to the received pixel position for retrieving from the captured image 

1 1 memory the captured image brightness value at the received pixel position and for compensating 

12 the retrieved captured imaged brightness value at the received pixel positions by the inverse of a 

13 response function of a light-sensing element having the sensitivity level corresponding to the 

14 received pixel position to provide a respective compensated brightness value at the pixel position 
pt5 received from the pixel position generator; 

Lfl6 (d) a compensated on-grid brightness value memory for storing respective compensated 

? TI captured image brightness values derived by the response function compensator; 
r jj8 (e) an off-grid position generator for providing the off-grid positions of the off-grid 

r|9 position array; 

liO (f) a pseudoinverse estimator receiving off-grid positions from the off-grid position 

;:|l generator and being responsive to an off-grid position received from the off-grid position 

22 generator for deriving from the compensated captured image brightness values in the 

23 compensated on-grid brightness value memory a respective estimated off-grid brightness value at 

24 the off-grid positions received from the off-grid position generator. 

1 123. The system of claim 122, wherein the output image generator comprises 
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2 (i) an off-grid brightness value memory for storing the respective estimated off- 

3 grid brightness values derived by the pseudoinverse estimator; 

4 (ii) an on-grid resampler for deriving from the estimated off-grid brightness 

5 values in the off-grid brightness value memory respective on-grid resampled brightness values at 

6 the pixel positions of the pixel position array; and 

7 (iii) an output image memory for storing the respective resampled on-grid 

8 brightness values derived by the on-grid resampler as pixel brightness values of the output 
q9 image, and wherein the on-grid resampler comprises: 

LfflO (1) a second pixel position generator for providing the pixel positions of 

|S | 1 the pixel position array; and 

'%2 (2) an interpolator receiving pixel positions from the pixel position 

i;J3 generator and being responsive to a pixel position received from the second pixel position 

TiM generator for deriving from the estimated off-grid brightness values in the off-grid brightness 

;:|5 value memory the resampled on-grid brightness value at the pixel position received from the 

16 second pixel position generator. 

1 124. The system of claim 122, wherein the response function compensator comprises a 

2 lookup table memory storing separate lookup table data representing the inverse of respective 

3 response functions of light-sensing elements having different ones of the plurality of sensitivity 

4 levels, and a mapper receiving from the captured image memory the captured image brightness 
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5 value at the pixel position received from the first pixel position generator for deriving therefrom 

6 a respective index for the lookup table data representing the inverse of the response function of a 

7 light-sensing element having the sensitivity level corresponding to the received pixel position, 

8 and providing a compensated on-grid brightness value corresponding to the index to the 

9 compensated on-grid brightness value memory. 

1 125. The system of claim 122, wherein the pseudoinverse estimator comprises means 

i=~2 responsive to an off-grid positions received from the off-grid position generator for retrieving 

! s fp from the compensated on-grid brightness value memory respective compensated on-grid 

brightness values at a predetermined number of pixel positions that are nearest neighbors to the 

^5 received off-grid position, a comparator for comparing each one of the respective compensated 

;=g6 on-grid brightness values at the nearest neighbor pixel positions to a predetermined low threshold 

!'U7 value and a predetermined high threshold value, an adder means for computing a sum of 

^8 compensated on-grid brightness values at the nearest neighbor pixel positions that are greater 

9 than the predetermined low threshold value and less than the predetermined high threshold value, 

10 and a divider for deriving the estimated off-grid brightness value at the received off-grid position 

1 1 by dividing the sum computed by the adder by the number of compensated on-grid brightness 

12 values included in the sum. 



NY02:24 1764.1 



- 150- 




FILE NO. A32708-070050.1 125 

1 126. The system according to claim 122, wherein the pixel positions provided by the 

2 pixel position generator are located at respective intersections of a plurality of regularly spaced 

3 pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to 

4 the pixel columns, and the off-grid positions provided by the off-grid position generator are 

5 located at respective intersections of a plurality of regularly spaced intermediate rows and a 

6 plurality of regularly spaced intermediate columns, each one of the intermediate rows being 

7 parallel to the pixel rows and extending medially between a respective adjacent pair of the pixel 
q8 rows, each intermediate column being parallel to the pixel columns and extending medially 

109 between a respective adjacent pair of pixel columns, and wherein the off-grid positions provided 

;: I0 by the off-grid position generator have coordinates (x',y ') where x' has values from 1 to xSize- 1 

"|1 and y' has values from 1 to y Size- 1, xSize being the dimension of the captured image in the 

iJ2 direction of the pixel rows and ySize being the dimension of the captured image in the direction 

fi3 of the pixel columns. 

1 127. The system of claim 126, wherein the predetermined sensitivity pattern of the array 

2 of light-sensing elements corresponds to a pixel position array having repetitively disposed 

3 groups of four nearest neighbor pixel positions, where each of the four nearest neighbor pixel 

4 positions in each group corresponds to a respective one of four predetermined sensitivity levels 

5 in a common predetermined positional order. 
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1 128. The system of claim 126, wherein the pseudoinverse estimator for estimating the 

2 respective brightness value at an off-grid position (x',y') received from the off-grid position 

3 generator comprises: 

4 (i) a memory for storing a 4 x 4 cubic interpolator kernel (J>; 

5 (ii) a comparator for comparing the compensated on-grid brightness value at each 

6 one of sixteen pixel positions (u,v) that are nearest neighbors to the received off-grid position 

7 (x',y ') with a predetermined low threshold value and a predetermined high threshold value; 
q8 (iii) first computing means for calculating the indices i and j of the 4x4 cubic 
Ln9 interpolation kernel (J> using the relations i = x'-u- 1 .5 and j = y v- 1 .5 for each one of the 
|S f 0 sixteen nearest neighbor pixel positions (u,v) having a compensated on-grid brightness value 

I(u,v) that is greater than the predetermined low threshold value and less than the predetermined 

r|2 high threshold value, where u has values from x'- 1 .5 to x'+l .5 and v has values from y'- 1 .5 to 

m y'+1.5; 

;;f 4 (iv) second computing means for calculating the quantity 

, 4>»/(*>v) 
(h 2 31 

ab 

1 5 for each compensated on-grid brightness value I(u 5 v) at the sixteen nearest neighbor pixel 

16 positions (u,v) that is greater than the predetermined threshold value and less than the 
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17 predetermined threshold value using the indices i and j calculated by the first computing means, 

1 8 where (f) mn is the value of the 4 * 4 interpolating kernel <|> at indices m and n; 

19 (v) third computing means for calculating the quantities for values of i and j 

20 calculated by the first computing means; 

21 (vi) first adder for adding the quantities 



iJ2 calculated by the second computing means to derive a first sum s; 

;: 23 (vii) second adder for adding the quantities cj>^. calculated by the third computing 

1^4 means to derive a second sum w; and 

■;;25 (viii) a divider for deriving the estimated off-grid brightness value at the received 

r36 off-grid position (x',y') by dividing the first sum s by the second sum w. 

1 129. The system according to claim 122, wherein the pixel positions provided by the 

2 pixel position generator are located at respective intersections of a plurality of regularly spaced 

3 pixel rows and a plurality of regularly spaced pixel columns, and the off-grid positions provided 

4 by the off-grid position generator are located at respective intersections of a plurality of regularly 

5 spaced intermediate rows and the pixel columns, and at respective intersections of a plurality of 

6 regularly spaced intermediate columns and the pixel rows, each one of the intermediate rows 
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7 being parallel to the pixel rows and extending medially between a respective adjacent pair of the 

8 pixel rows, each intermediate column being parallel to the pixel columns and extending medially 

9 between a respective adjacent pair of the pixel columns, the off-grid positions provided by the 

10 off-grid position generator having coordinates (x'y') where x' has values from 1 to xSize- 1 and 

1 1 y' has values from 0.5 to ySize-0.5 for off-grid positions located at respective intersections of 

12 the pixel rows and the intermediate columns, and where x' has values from 0.5 to y Size- 0.5 and 

13 y' has values from 1 to ySize- 1 for off-grid positions located at respective intersections of the 
i= y pixel columns and the intermediate rows, xSize being the dimension of the captured image in the 
LB5 direction of the pixel rows and ySize being the dimension of the captured image in the direction 
: = f6 of the pixel columns. 

i-il 130. The system of claim 129, wherein the predetermined sensitivity pattern of the array 

rU2 of light-sensing elements has first and second predetermined sensitivity levels, and adjacent pixel 

[:^3 positions in each pixel row and adjacent pixel positions in each pixel column correspond to 

4 different ones of the first and second predetermined sensitivity levels. 

1 131. The system of claim 129, wherein the pseudoinverse estimator for estimating the 

2 respective off-grid brightness value at an off-grid position (x\y') received from the off-grid 

3 position generator comprises: 

4 (i) a memory for storing a 4 x 4 cubic interpolation kernel <j); 
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5 (ii) first computing means responsive to an off-grid position (x' 3 y') received from the off- 

6 grid position generator for computing coordinates (u 5 v) using the relations u=x'+0.5 i-0.5j and 

7 v=y'-1.5+0.5i+0.5j, where i and j are indices of the 4 x 4 cubic interpolation kernel (j), and each 

8 one of the indices i and j has values from 0 to 3; 

9 (iii) means responsive to each pair of coordinates (u,v) computed by the first computing 

10 means for determining whether the coordinates (u,v) are those of a pixel position; 

1 1 (iv) a comparator responsive to the coordinates (u,v) being coordinates of a pixel position 
jA2 for comparing the compensated on-grid brightness value I(u,v) at the pixel position (u 5 v) with a 
Lif3 predetermined low threshold value and a predetermined high threshold value; 

i = f 4 (v) second computing means responsive to the coordinates (u ? v) being the coordinates of 

;%5 a pixel position and the compensated on-grid brightness value I(u,v) at the pixel position (u ? v) 

r|6 being greater than a predetermined low threshold value and less than a predetermined high 

\W7 threshold value for calculating the quantity 

ab 

18 where the values of i and j are those used by the first computing means to calculate the 

19 coordinate (u ? v), and (j) mn is the value of the 4x4 cubic interpolation kernel c|> at indices m and n; 

20 (vi) third computing means for calculating the quantity c|)^ for values of i and j used by 

21 the first computing means to calculate pixel position coordinates (u,v) at which the compensated 
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22 on-grid brightness value I(u,v) is greater than the predetermined low threshold value and less 

23 than the predetermined high threshold value; 

24 (vii) a first adder for adding the quantities 



25 calculated by the second computing means to derive a first sum s; 

r|6 (viii) a second adder for adding the quantities (J)^. calculated by the third computing 

'Ml means to derive a second sum w; and 

j: |8 (ix) a divider for deriving the estimated off-grid brightness value at the off-grid position 

! = 29 (x',y') received from the off-grid position generator by dividing the first sum s by the second 

□jO sumw. 

1 132. The system of claim 131, wherein the 4x4 cubic interpolation kernel (J) has the 

2 form: 
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rjl 133. The system of claim 122, wherein the pixel positions provided by the pixel position 

! S U2 generator are located at respective regularly spaced pixel positions in a linear pixel position array 

] :^3 and the off-grid positions provided by the off-grid position generator are each located midway 

4 between a respective adjacent pair of pixel positions of the linear pixel position array, and 

5 wherein the predetermined sensitivity pattern of the light-sensing elements has a first and a 

6 second predetermined sensitivity level, and adjacent pixel positions of the linear pixel position 

7 array correspond to different ones of the first and second predetermined sensitivity levels. 
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1 134. The system of claim 133, wherein the pixel positions provided by the pixel position 

2 generator has values from 0.5 to xSize-0.5, where xSize is the dimension of the captured line 

3 image, and the off-grid positions provided by the off-grid position generator have values from 2 

4 toxSize-2. 

1 135. The system of claim 134, wherein the pseudo inverse estimator for estimating the 

2 respective brightness values at an off-grid position x' received from the off-grid position 
; ==3 generator comprises: 

LIT4 (i) a memory for storing a 4 x 1 Gaussian interpolation kernel G; 

\ s *5 (ii) a comparator for comparing the compensated on-grid brightness value I(k) at each one 

:= L;6 of four pixel positions k that are nearest neighbors to the received off-grid position x' with a 

predetermined low threshold value and a predetermined high threshold value, where k has values 

Hj8 from x'- 1.5 tox'+1.5; 

£39 (iii) first computing means for calculating the quantity I(k)G(k-x') for each one of the 

10 compensated on-grid brightness values I(k) at the four nearest neighbor pixel positions k that is 

1 1 greater than the predetermined low threshold value and less than the predetermined high theshold 

12 value, where G(k-x') is the value of the Gaussian interpolation kernel G at position (k-x'); 

13 (iv) a first adder for adding the quantities I(k)G(k-x') for all values of I(k) greater than 

14 the predetermined low threshold value and less than the predetermined high threshold value to 

1 5 derive a first sum p; 
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16 (v) a second adder for adding the Gaussian interpolation kernel values G(k-x') for all 

17 values of k where I(k) is greater than the predetermined low threshold value and less than the 

1 8 predetermined high threshold value to derive a second sum q; and 

19 (vi) a divider for deriving the estimated off-grid brightness value B(x') at the off-grid 

20 position x' received from the off-grid position generator. 

1 136. The apparatus of claim 125, 128, 131 or 135, wherein the predetermined low 

,==§2 threshold value is the noise level of a light-sensing element having a sensitivity level 

yi3 corresponding to the pixel position of the compensated on-grid brightness value being compared 

i B ^4 therewith compensated by the inverse of the response function of the light-sensing element, and 

= LZ5 the predetermined high threshold value is the saturation or near saturation brightness value of a 

□6 light-sensing element having the sensitivity level corresponding to the pixel position of the 

fy7 compensated on-grid brightness value being compared therewith. 

1 137. The apparatus of claim 123, wherein the pixel positions of the pixel position array 

2 provided by the second pixel position generator are located at respective regularly spaced pixel 

3 positions of a linear pixel position array, and the off-grid positions of the off-grid position array 

4 provided by the off-grid position generator are each located between a respective adjacent pair of 

5 pixel positions of the linear pixel position array. 
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1 138. The apparatus of claim 123, wherein the pixel positions of the pixel position array 

2 provided by the second pixel position generator are located at respective intersections of a 

3 plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the 

4 pixel rows being orthogonal to the pixel columns, and wherein the off-grid positions of the off- 

5 grid position array provided by the off-grid position generator are located at respective 

6 intersections of a plurality of regularly spaced intermediate rows and a plurality of regularly 

7 spaced intermediate columns, each intermediate row being parallel to the pixel rows and 

r -^8 extending between a respective adjacent pair of the pixel rows and each intermediate column 

jl9 being parallel to the pixel columns and extending between a respective adjacent pair of the pixel 

! s f€ columns. 

^1 1 39. The apparatus of claim 123, wherein the pixel positions of the pixel position array 

f y2 provided by the second pixel position generator are located at respective intersections of a 

□3 plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the 

4 pixel rows being orthogonal to the pixel columns, and wherein the off-grid positions of the off- 

5 grid position array provided by the off-grid position generator are located at respective 

6 intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one 

7 of the intermediate pixel columns being parallel to the pixel columns and extending between a 

8 respective adjacent pair of pixel columns. 
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1 140. The apparatus of claim 137, 138 or 139, wherein the interpolator includes means for 

2 shifting the off-grid position array to coincide with the pixel position array, whereby the 

3 resampled on-grid brightness value at each pixel position of the pixel position array is equal to 

4 the estimated off-grid brightness value at a coincident off-grid position of the shifted off-grid 

5 position array. 

1 141 . The apparatus of claim 138, wherein the interpolator derives a respective resampled 
; ™2 on-grid brightness value at a pixel position received from the second pixel position generator by 
Ln3 computing the product of respective estimated off-grid brightness values at a predetermined 
^4 number of off-grid positions that are nearest neighbors to the received pixel position and an 

interpolation kernel having the same dimensions as the predetermined number of off-grid 

□6 positions. 

y 1 142. The apparatus of claim 141, wherein the plurality of intermediate rows each extend 

2 medially between a respective adjacent pair of pixel rows and the plurality of regularly spaced 

3 intermediate columns each extend medially between a respective adjacent pair of pixel columns, 

4 and wherein the pixel positions provided by the second pixel position generator have coordinates 

5 (x,y) where x has values from 2.5 to xSize-2.5 and, y has values from 2.5 to ySize-2.5, xSize is 

6 the dimension of the captured image in the direction of the pixel rows and ySize is the dimension 

7 of the captured image in the direction of the pixel columns. 
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1 143. The apparatus of claim 142, wherein the interpolator is responsive to a pixel position 

2 (x,y) received from the second pixel position generator for deriving a respective resampled on- 

3 grid brightness value in accordance with the relation 

Kx-ly-2) = £ E 5(x-1.5 + z,y-1.5 + jft 

i=o j=0 U 

r~4 where I(x-2 ? y-2) is the resampled on-grid brightness value at pixel position (x-2, y-2), 

;.n5 B(x- 1 .5+i, y- 1 .5+j) is the estimated off-grid brightness value at off-grid position 

^6 (x- 1 .5+i, y- 1 .5+j), (foj is the value of a 4 x 4 cubic interpolation kernel <j> at indices i and j, and 

= ~7 indices i and j each has values from 0 to 3. 

fUl 144. The system of claim 123, wherein the pixel positions provided by the second pixel 

y 2 position generator are located at respective intersections of a plurality of regularly spaced pixel 

3 rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the 

4 pixel columns, and wherein the off-grid positions provided by the off-grid position generator are 

5 located at respective intersections of a plurality of regularly spaced intermediate rows and the 

6 pixel columns, and at respective intersections of a plurality of regularly spaced intermediate 

7 columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows 

8 and extending medially between a respective adjacent pair of the pixel rows, each intermediate 
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9 column being parallel to the pixel columns and extending medially between a respective adjacent 

10 pair of the pixel columns, the pixel positions provided by the second pixel position generator 

1 1 having coordinates (x,y) where x has values from 2.5 to xSize-2.5, y has values from 2.5 to 

12 y Size- 2.5, xSize is the dimension of the captured image in the direction of the pixel rows and 

13 ySize is the dimension of the captured image in the direction of the pixel columns. 

1 145. The system of claim 144, wherein the interpolator is responsive to a pixel position 

,=b2 received from the second pixel position generator for deriving a respective resampled on-grid 

I jl3 brightness value in accordance with the relation 

r I 3 3 

t I(x-2y-2) = £ £ 5(x + 0.5/-0^,^-1.5 + 0.5z + 0.^)A , 

l l! i=o j-o lJ 

rU4 where I(x-2,y-2) is the resampled brightness value at pixel position (x-2, y-2), 

y5 B(x-K).5i-0.5j, y- 1.5+0.5i+0.5j) is the off-grid brightness value at off-grid position 

6 (x+0.5i-0.5j, y- 1.5+0. 5i+0.5j), (foj is the value of a 4 x 4 cubic interpolation kernel (J) at indices i 

7 and j, and the indices i and j each have values from 0 to 3. 
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146. The method of claim 128 or 143, wherein the 4 x 4 cubic interpolation kernel (() has 
the form: 
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147. The apparatus of claim 141, wherein the interpolation kernel is a 2-dimensional 
Gaussian kernel. 

148. The system of claim 147, wherein the 2-dimensional Gaussian kernel has the form: 
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149. The system of claim 141, wherein the interpolation kernel is a bi-cubic B-spline 
interpolation filter kernel. 
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150. The system of claim 149, wherein the bi-cubic B-spline interpolation filter kernel 
has the form: 



0.0043 


0.0100 


0.0100 


0.0043 
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151. The apparatus of claim 141, wherein the interpolation kernel is a bi-linear 
interpolation filter kernel. 

152. The system of claim 151, wherein the bi-linear interpolation kernel has the form: 



0.5 


0.5 


0.5 


0.5 


► 



i 



153. The system of claim 137, wherein the off-grid positions provided by the off-grid 
position generator are each located midway between a respective adjacent pair of pixel positions 
of the linear pixel position array, the pixel positions provided by the second pixel position 
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4 generator having coordinate x, where x has values from 3.5 to xSize-3.5 and xSize is the 

5 dimension of the captured line image. 

1 1 54. The system of claim 1 53, wherein the interpolator is responsive to a pixel position 

2 received from the second pixel position generator for deriving a respective resampled on-grid 

3 position brightness value in accordance with the relation 

*=1.5 

4 /(x-3) = £ B(x+kMk) 9 

*=-1.5 

! s fj5 where I 0 (x-3) is the resampled on-grid brightness value at pixel position (x-3), B(x+k) is the 
^6 estimated off-grid brightness value at off-grid position (x+k), k is a position of a 4 x 1 cubic 
^7 interpolation kernel i|j 5 i|/(k) is the value of the cubic interpolation kernel i|r at position k, and the 
p8 position k has values from - 1 .5 to 1.5. 

1 1 55. The system of claim 154, wherein the 4 x 1 cubic interpolation kernel i|j has the 

2 form: 
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1 156. A system for obtaining a relatively high dynamic range image of a scene using a 

2 relatively low dynamic range image sensor adapted to be exposed to incident light from the scene 

3 for capturing an image thereof comprising: 

4 (a) an image sensor having a multiplicity of light-sensing elements in an array, each one 

5 of the light-sensing elements having a particular one of a plurality of sensitivity levels to incident 

6 light in accordance with a predetermined sensitivity pattern for the array of light-sensing 

q7 elements and having a response function, each one of the light-sensing elements in response to 

Ln8 incident light from the scene producing a captured image brightness value at a corresponding one 

!= "9 of a multiplicity of pixel positions of a pixel position array, whereby each one of the multiplicity 

;%0 of pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light- 

r| 1 sensing elements ; 

rtU2 (b) a captured image memory for storing respective captured image brightness values 

;:| 3 produced by the light-sensing elements at pixel positions of the pixel position array; 

14 (c) a pixel position generator for providing the pixel positions of the pixel position array; 

15 (d) an on-grid brightness value generator receiving the pixel positions from the pixel 

16 position generator and being responsive to a pixel position received from the pixel position 

1 7 generator for deriving from the captured image brightness values in the captured image memory 

18 a respective output image brightness value at the received pixel position, the respective output 
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19 image brightness value being derived from a corresponding plurality of captured image 

20 brightness values; and 

21 (e) an output image memory for storing the respective output image brightness value at 

22 each pixel position received from the pixel position generator derived by the on-grid brightness 

23 value generator. 

1 157. The system of claim 156, wherein the array of light-sensing elements of the image 

sensor is a two-dimensional array and the pixel position array is a linear array having regularly 

LF3 spaced pixel positions in a linear pixel position row. 

^1 1 58. The system of claim 156, wherein the array of light-sensing elements of the image 

r-2 sensor is a two-dimensional array and the pixel position array is a two-dimensional array having 

i y 

j = U3 pixel positions located at respective intersections of a plurality of regularly spaced pixel rows and 

|;=}4 a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel 

5 columns, and wherein the pixel positions provided by the pixel position generator have 

6 coordinates (x,y), where x has values from 2.5 to xSize-2.5, y has values from 2.5 to ySize-2.5, 

7 and xSize and ySize are the dimensions of the captured image in the directions of the pixel rows 

8 and the pixel columns, respectively. 
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1 159. The system of claim 158, wherein the on-grid brightness value generator is 

2 responsive to a pixel position (x,y) received from the pixel position generator for deriving from 

3 the captured image brightness values in the captured image memory a respective output image 

4 brightness value at the received pixel position in accordance with the relation 

I o (x-2y-2) = £ E 7(x-2 + i, y-2 + jft 

i=o j=0 

where I c (x - 2 + i ? y - 2 + j) is the captured image brightness value at pixel position 
i fk) (x - 2 + i ? y - 2 + j), (^j is the value of an on-grid interpolation filter kernel (j) at indices i and j, 

indices i and j each have values from 0 to 4, and I 0 (x - 2, y -2) is the output image brightness 
i= ij8 value at pixel position (x - 2, y - 2). 

ml 160. A system for obtaining a relatively high dynamic range image of a scene using a 

=132 relatively low dynamic range image sensor adapted to be exposed to incident light from the scene 

3 for capturing an image thereof comprising: 

4 (a) an image sensor having a multiplicity of light-sensing elements in an array, each one 

5 of the light-sensing elements having a particular one of a plurality of sensitivity levels to incident 

6 light in accordance with a predetermined sensitivity pattern for the array of light-sensing 

7 elements and having a response function, each light-sensing element in response to incident light 

8 from the scene producing a captured image brightness value at a corresponding one of a 
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9 multiplicity of pixel positions of a pixel position array, whereby each one of the multiplicity of 

10 pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light- 

1 1 sensing elements; 

12 (b) a captured image memory for storing the captured image brightness values produced 

13 by the light-sensing elements at pixel positions of the pixel position array; 

14 (c) a pixel position generator for providing the pixel positions of the pixel position array; 

15 (d) an on-grid brightness value generator receiving pixel positions from the pixel position 
,46 generator and being responsive to a pixel position received from the pixel position generator for 
;j|7 deriving from the captured image brightness values in the captured image memory a respective 
HI 8 on-grid brightness value at the received pixel position, the respective on-grid brightness value 

9 being derived from a corresponding plurality of captured image brightness values; 
; 50 (e) a response function compensator for compensating the respective on-grid brightness 

rOl value at each received pixel position derived by the on-grid brightness value generator by the 

Q2 inverse of a combined response function of the light-sensing elements to provide a respective 

23 output image brightness value at each pixel position received from the pixel position generator, 

24 the combined response function being the sum of weighted response functions of light-sensing 

25 elements having different ones of the plurality of sensitivity levels, each one of the weighted 

26 response functions being a response function of a light-sensing element having a respective one 

27 of the plurality of sensitivity levels multiplied by a predetermined weighting factor for the 

28 respective one of the plurality of sensitivity levels; and 
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29 (f) an output image memory for storing the respective output image brightness value at 

30 each received pixel positions provided by the response function compensator. 

1 161. The system of claim 160, wherein the response function compensator comprises a 

2 lookup table memory storing data representing the inverse of the combined response function of 

3 the light-sensing elements, and a mapper receiving on-grid brightness values from the on-grid 

4 brightness value generator and being responsive to an on-grid brightness value at a pixel position 
q5 received from the pixel position generator for deriving a respective index for the lookup table 

:. : t 

iJl6 memory and providing a data value in the lookup table memory corresponding to the respective 

V Z7 index, the data value being the output image brightness value at the pixel position received from 

fn 

>|8 the pixel position generator. 

FUl 162. The system of claim 160, wherein the array of light-sensing elements of the image 

;~ 2 sensor is a two-dimensional array for capturing two-dimensional images, and the pixel position 

3 array is a two-dimensional array having pixel positions located at respective intersections of a 

4 plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the 

5 pixel rows being orthogonal to the pixel columns, and wherein the pixel positions provided by 

6 the pixel position generator having coordinates (x,y), where x has values from 2.5 to xSize-2.5, 

7 y has values from 2.5 to ySize-2.5, and xSize and ySize are the dimensions of the captured 

8 image in the directions of the pixel rows and the pixel columns, respectively. 
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1 163. The system of claim 162, wherein the on-grid brightness value generator is 

2 responsive to a pixel position received from the pixel position generator for deriving from the 

3 captured image brightness values in the captured image memory a respective on-grid brightness 

4 value at the received pixel position in accordance with the relation 

/(x-2j,-2) = £ E 7(*-2 + Uy-2 + j)<b if 

5 where I c (x - 2 + i, y - 2 + j) is the captured image brightness value at pixel position 

6 (x - 2 + i 5 y - 2 + j) ? 4>ij is the value of a on-grid interpolation filter kernel (J) at indices i and j, the 

7 indices i and j each have values from 0 to 4, and I(x - 2, y -2) is the on-grid brightness value at 

8 pixel position (x - 2, y - 2). 
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2 



164. The method of claim 159 or 163, wherein the on-grid interpolation filter kernel (J) 
has the form: 
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-0.618 


9.001 


19.238 


9.001 


-0.618 


7 r 




0.043 


-0.618 


-1.322 


-0.618 


0.043 


8 


► 











11 165. The system of claim 158, wherein the array of light-sensing elements of the image 

12 sensor is a linear array for capturing a line image, and the pixel position array is a linear array 
=3 having regularly spaced pixel positions in a linear pixel row. 



! U1 166. A system for obtaining a relatively high dynamic range image of a scene using a 

;;i2 relatively low dynamic range image sensor adapted to be exposed to incident light from the scene 

3 for capturing an image thereof comprising: 

4 (a) an image sensor having a multiplicity of light-sensing elements in an array, each one 

5 of the light-sensing elements having a particular one of a plurality of sensitivity levels to incident 

6 light in accordance with a predetermined sensitivity pattern for the array of light-sensing 

7 elements and having a respective response function, each light-sensing element in response to 

8 incident light from the scene producing a captured image brightness value at a corresponding one 
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9 of a multiplicity of pixel positions of a pixel position array, whereby each one of the multiplicity 

10 of pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light- 

1 1 sensing elements; 

12 (b) a captured image memory for storing the captured image brightness values produced 

13 by the light-sensing elements at pixel positions of the pixel position array; 

14 (c) a first pixel position generator for providing the pixel positions of the pixel position 

15 array; 

r=|6 (d) a sensitivity pattern memory for storing data indicative of the sensitivity level 

\W7 corresponding to each one of the first pixel positions of the pixel position array; 

H;8 (e) a response function compensator receiving pixel positions from the first pixel position 

= I9 generator, data indicative of the sensitivity levels corresponding to the received pixel positions 

;;|0 from the sensitivity pattern memory and captured image brightness values at the received pixel 

TBI positions from the captured image memory, and being responsive to a pixel position received 

i|2 from the first pixel position generator and data indicative of the sensitivity level corresponding to 

23 the received pixel position for retrieving from the captured image memory the captured image 

24 brightness value at the received pixel position and for compensating the retrieved captured image 

25 brightness value at the received pixel position by the inverse of the response function of a light- 

26 sensing element having the sensitivity level corresponding to the received pixel position to 

27 provide a respective compensated brightness value at the pixel position received from the first 

28 pixel position generator; 
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29 (f) a compensated on-grid brightness value memory for storing respective compensated 

30 captured image values at the pixel positions received from the first pixel position generator; 

31 (g) a second pixel position generator for providing the pixel positions of the pixel 

32 position array; 

33 (h) an on-grid brightness value generator receiving pixel positions from the second pixel 

34 position generator and being responsive to each pixel position received from the second pixel 

35 position generator for deriving from the captured image brightness values in the captured image 
|46 memory a respective output image brightness value at each received pixel position; and 

L37 (i) an output image memory for storing the respective output image brightness value at 

!= |8 each pixel position received from the second pixel position generator. 

□1 167. The system of claim 166, wherein the response function compensator comprises a 

tU2 lookup table memory storing separate lookup table data representing the inverse of respective 

;:f3 response functions of light-sensing elements having different ones of the plurality of sensitivity 

4 levels, and a mapper receiving captured image brightness values from the captured image 

5 memory and being responsive to a captured image brightness value at a pixel position received 

6 from the first pixel position generator for deriving a respective index for the lookup table 

7 memory, and retrieving therefrom a compensated on-grid brightness value corresponding to the 

8 respective index from lookup table data representing the inverse of the response function of a 

9 light-sensing element having the sensitivity level corresponding to the received pixel position, 
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10 the retrieved compensated on-grid brightness value being provided to the compensated on-grid 

1 1 brightness value memory. 

1 168. The system of claim 166, wherein the array of light-sensing elements of the image 

2 sensor is a two-dimensional array, and the pixel position array is a two-dimensional array having 

3 pixel positions located at respective intersections of a plurality of regularly spaced pixel rows and 

4 a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel 
=|5 columns. 

~1 169. The system of claim 166, wherein the array of light-sensing elements of the image 

^2 sensor is a linear array for capturing a line image and the pixel position array is a linear array 

33 having respective regularly spaced pixel positions of a linear pixel row, and wherein the pixel 

U4 positions provided by the pixel position generator have coordinates x, where x has values from 

!f5 2.5 to xSize-2.5 and xSize is the dimension of the captured line image. 

1 170. The system of claim 169, wherein the on-grid brightness value generator is 

2 responsive to a pixel position x received from the second pixel position generator for deriving 

3 from the captured image brightness values in the captured image memory a respective output 

4 image brightness value at the received pixel position, the on-grid brightness value generator 

5 comprising: 
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6 (i) a memory for storing a 5 * 1 Gaussian interpolation filter kernel G; 

7 (ii) a comparator for comparing each one of five compensated on-grid brightness values 

8 I(k) at pixel positions k having values from x-2 to x+2 with a predetermined low threshold value 

9 and a predetermined high threshold value; 

10 (iii) a multiplier for calculating a quantity I(k)G(k-x) for each value of k at which I(k) is 

1 1 greater than the predetermined low threshold value and less than the predetermined high 

12 threshold value, where G(k-x) is the value of the Gaussian interpolation kernel G at position 
;:|3 (k-x); 

^ffi4 (iv) a first adder for adding the quantities I(k)G(k-x) calculated by the multiplier for each 

l= |5 value of k where I(k) is greater than the predetermined low threshold value and less than the 

I j6 predetermined high threshold value to derive a first sum p; 

i;17 (v) a second adder for adding the values G(k-x) of the Gaussian interpolation kernel in 

l ^8 the memory for each value of k where I(k) is greater than the predetermined low threshold value 

;J9 and less than the predetermined high threshold value to derive a second sum q; and 

20 (vi) a divider for deriving a respective output image brightness value I 0 (x) at the position 

21 x received from the second pixel position generator by dividing the sum p by the sum q. 

1 171. The system of claim 1 70, wherein the predetermined low threshold value is the 

2 noise level of a light-sensing element having the sensitivity level corresponding to the pixel 

3 position of the compensated on-grid brightness value being compared therewith compensated by 
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4 the inverse of the response function of the light-sensing element, and the predetermined high 

5 threshold value is the saturation or near saturation brightness value of a light-sensing element 

6 having the sensitivity level corresponding to the pixel position of the compensated on-grid 

7 brightness value being compared therewith. 

1 172. The system of claim 170, wherein the 5 x 1 Gaussian interpolator filter kernel has 

2 the form: 
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